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EXECUTIVE  SUMMARY 


Multinational  Limited  Objective  Experiment  (MN  LOE)  II  was  the  second  event 
in  a  series  of  four  multinational  experiments.  This  process-refinement  experiment 
examined  how  to  build  a  multinational  operational  net  assessment  (ONA)  using  a 
distributed,  collaborative  information  environment  (CIE),  as  well  as  information  release 
issues  among  coalition  partners.  Additionally,  it  assessed  the  viability  of  the 
multinational  information  sharing  for  the  allies  and  coalition  partners  (MNISACP) 
concept  of  operations  developed  by  the  USJFCOM  J6.1  MN  LOE  I,  conducted  in 
November  2001,  examined  how  a  combined  joint  force  headquarters  might  plan  to 
conduct  rapid,  decisive  operations  in  a  distributed,  collaborative  infonnation  environment 
with  coalition  partners.  The  remaining  two  experiments  will  build  upon  the  lessons 
learned  from  MN  LOE  I  and  II  and  will  address  combined  planning  and  execution.  This 
series  of  experiments  contributes  to  USJFCOM’s  transformation  strategy  by  feeding  its 
concept  development  and  prototyping  paths.  An  overarching  objective  of  the 
multinational  experiment  series  is  to  include  multinational  partners  in  Joint  Futures  Lab 
and  Distributed  Continuous  Experimentation  Environment  events  that  promote 
multinational  agency  participation. 

Results  of  multinational  experimentation  will  support  further  development  of  a 
standing  joint  force  headquarters  and  will  provide  data  for  information  sharing,  multilevel 
security,  and  collaborative  ONA  development  to  both  the  NATO  Concept  Development 
and  Experimentation  (CDE)  Working  Group  and  to  the  Multinational  Interoperability 
Council  (MIC)  Working  Group.  The  results  also  will  provide  evidence  that  addresses  the 
task  assigned  by  the  Chairman,  Joint  Chiefs  of  Staff,  to  develop  a  MNIS  operational 
concept  and  associated  architecture. 

The  participants  of  MN  LOE  II — Australia,  Canada,  Germany,  the  United 
Kingdom,  and  the  United  States — are  members  of  the  MIC,  which,  along  with  France, 
unanimously  designated  the  experiment  as  a  Level  2  CDE  event2.  NATO  CDE/Supreme 
Allied  Commander  Atlantic  (SACLANT)  played  the  role  of  a  separate  nation.  France, 
the  sixth  MIC  member  nation,  actively  observed  the  experiment  in  an  effort  to  leam  the 
CDE  process  and  to  prepare  for  participation  in  MN  Experiment  III.  The  experiment  was 
conducted  in  a  distributed  environment.  The  U.S.  planning  cells  were  located  within  J9 
Joint  Experimentation  in  Suffolk,  Virginia,  and  the  Australian,  UK,  German,  and 
Canadian  cells  were  located  in  Fem  Hill  Park,  Portsdown  West,  Ottobrunn,  and  Ottawa, 
respectively.  The  Combined  Federated  Battle  Laboratory  (CFBL),  a  dedicated  wide-area 
network  for  experimentation,  provided  connectivity  for  the  MN  LOE.  Participating 


1  See  Appendix  J  for  an  overview  of  the  ONA  and  MNIS  concepts. 

“  In  a  Level  2  CDE  event  MIC  member  nations  participate  as  guests  in  other  members’  CDE 
initiatives.  A  lead  nation  provides  the  CDE  infrastructure  and  secretarial  support  that  “host” 
supporting  member  nations,  who  meet  their  own  in-country  costs  and  who  help  to  shape  the 
event,  at  the  request  of  the  lead  nation. 
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nations  took  an  active  role  in  the  design  and  execution  of  the  experiment.  Three 
scheduled  and  two  ad  hoc  workshops  introduced  the  U.S.  ONA  process  prior  to  MN  LOE 
II  execution  in  Suffolk,  Va. 

The  MN  LOE  II  used  a  pre-crisis  scenario,  set  in  a  peaceful  area  in  2010.  Amid 
an  ongoing  regional  military  and  political  dialogue,  the  regional  combatant  commander 
decided  to  watch  closely  the  activities  of  two  countries  in  the  area.  Potential  coalition 
partners  were  consulted,  and  led  by  the  United  States,  five  countries  agreed  to  develop  an 
MN  ONA  based  on  national  ONAs.  National  headquarters  began  collaboration  with  the 
regional  combatant  commander’s  standing  joint  force  headquarters  to  develop  an  MN  ONA. 

Experiment  Objective  1  was  to  identify  and  assess  the  ability  of  national 
headquarters  to  develop  a  distributed  multinational  ONA  database.  The  MN  LOE  II 
demonstrated  that  a  multinational  coalition  could  successfully  conduct  the  U.S.  ONA 
process  in  a  distributed  environment.  While  the  ONA  process  was  not  fully  exercised,  all 
participants  found  the  U.S.  approach  of  system-of-systems  analysis  of  the  adversary, 
ourselves,  and  the  operational  environment  to  be  powerful.  While  the  tools  require 
continued  development,  and  while  disclosure  policies  may  hamper  achievement  of  a  full 
multinational  capability,  all  participants  support  aggressive  pursuit  of  a  multinational  ONA. 

Experiment  Objective  2  was  to  identify  and  assess  collaboration  and  infonnation 
sharing  across  different  security  domains.  This  construct  was  subject  to  the  current  too- 
general  and  too-encompassing  infonnation-sharing  rules  that  constrain  the  ability  to  share 
information  in  pre-crisis  situations.  Such  restrictions  are  detrimental  to  building  trust 
within  the  coalition  and  to  developing  a  collective  understanding.  To  mitigate  the  fear 
and  perceived  risk  behind  current  information-sharing  rules,  the  potential  benefits  of 
collaboration  require  a  real  change  in  policy. 

MN  LOE  II  demonstrated  the  first  coalition  attempt  to  build  an  ONA  in  a 
distributed  environment.  The  experiment  also  examined  the  MNIS  operational  concept 
of  information  sharing  across  security  domains.  Finally,  MN  LOE  II  served  as  a  venue 
for  exploring  topics  of  national  interest.  The  feedback  from  all  participants  provided 
many  valuable  recommendations  that  ultimately  will  benefit  future  LOEs  and  other 
related  projects.  Experiment  analysts  from  all  nations  independently  evaluated  the 
experiment  data  and  presented  corroborating  results  at  the  post-event  workshop. 

Many  lessons  learned  and  recommendations  resulted  from  MN  LOE  II.  See 
Section  V  for  detailed  discussions  of  experimentation  lessons  learned. 

Eight  key  lessons  learned  may  enhance  future  experimentation: 

1.  Succinct  definitions  are  needed  for  all  experiment  objectives  and  issues. 

2.  Experiment  analysts  and  concept  writers  from  all  the  participating  nations 
must  be  directly  involved  in  the  experiment  planning  and  design  from  the 
early  stages  of  the  experiment  development. 
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3.  Any  changes  to  the  experiment  design  during  experiment  execution  must 
continue  to  be  coordinated  with  experiment  analysts  and  partner  nations  to 
ensure  conditions  that  produce  valid  and  reliable  findings. 

4.  Time  management  issues  must  be  considered  during  experiment  planning  to 
ensure  sufficient  time  is  allotted  to  study  all  critical  concepts  under 
investigation.  Furthermore,  earlier  stabilization  of  the  player  list  will  preclude 
many  of  the  time  constraint  issues. 

5.  Training  should  provide  in-depth  knowledge  of  essential  concepts  so  that 
participants  may  use  tools  and  may  transfer  knowledge  among  themselves 
during  the  experiment.  A  practice  walk-through  is  essential  to  give  players  the 
“big  picture”  and  to  teach  a  series  of  unified  tasks  rather  than  disconnected  steps. 

6.  A  robust  collaboration  concept  of  operations  or  tactics,  techniques,  and 
procedures  is  needed.  Future  experiment  events  will  require  more  thought  and 
consideration  of  collaboration. 

7.  Experiment  planning  sessions  should  be  designed  around  the  use  of  a 
collaborative  tool  or  teleconference  to  allow  for  all  multinational  experiment 
leads  to  be  more  directly  involved  with  the  planning  process.  All  key  U.S. 
participants  or  their  representatives  should  be  required  to  attend  IPT  meetings. 

8.  Robust  experimentation  with  MN  partners  will  require  a  boost  in  network 
bandwidth.  Actions  must  be  initiated  now  to  boost  bandwidth  availability. 

A  select  group  of  former  general  and  flag  officers,  known  as  senior  concept 
developers  (SCDs),  participated  in  a  variety  of  activities  as  a  source  of  experience  and 
knowledge  that  contributes  to  the  growing  understanding  of  the  concepts  examined  in  the 
experiment.  The  group’s  three  key  recommendations  were  deemed  essential  to  the 
development  of  the  concepts: 

1.  SCDs  urged  the  use  of  a  real-world  scenario  for  the  next  multinational 
experiment  (MNE),  scheduled  for  February  2004.  They  unanimously  and 
emphatically  endorsed  the  importance  and  value  of  this  recommendation, 
which  also  offers: 

□  The  addition  of  significant  granularity  to  the  ONA  database 

□  Greater  stress  placed  on  the  ONA  process  and  MNIS  hypothesis 

□  Increased  political  support  and  multinational  interagency  participation 

□  The  development  of  real-world  nodes,  links,  actions,  and  resources,  as 
well  as  the  generation  of  an  effects  plan  that  may  be  used  for  both 
experimentation  and  ongoing  stability  operations. 

2.  The  SCDs  recommended  that  the  commander,  USJFCOM,  should  advise  the 
Department  of  Defense  (DoD)  to  review  its  information- sharing  policy, 
practices,  and  procedures,  which  do  not  reflect  the  demanding  requirements  of 
the  information  age.  While  every  nation  would  have  a  foreign  disclosure 
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policy  regarding  highly  classified  information,  these  DoD  directives  should  be 
revised  to  emphasize  the  idea  of  “withholding  information  by  exception”  in  a 
coalition  environment. 

3.  The  SCDs  recommended  the  development  of  a  knowledge  white  paper  that 
would  generate  further  study,  debate,  and  exploration  of  this  critical  aspect  of 
future  joint  and  combined  operations.  Although  current  joint  force  concepts 
emphasize  the  value  of  and  necessity  for  knowledge,  a  distinct  “knowledge 
concept”  does  not  exist.  A  knowledge  white  paper  should  define  knowledge 
and  should  address  the  supporting  elements  of  knowledge  superiority, 
management,  readiness,  and  warfare. 

Although  any  experiment  offers  the  potential  to  examine  many  issues,  all  of  the 
partner  nations  of  the  MN  LOE  II  agreed  upon  the  critical  operational  issues  (COIs) 
necessary  to  explore  the  main  experiment  objectives.  This  table  depicts  the  findings  of 
the  experiment,  grouped  under  their  related  COIs. 
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EXPERIMENT  OVERVIEW 


A.  INTRODUCTION/PURPOSE 


Multinational  Limited  Objective  Experiment  (MN  LOE)  II  is  the  second  experiment  in  a 
series  of  four.  MN  LOE  I,  conducted  in  November  2001,  examined  the  way  a  combined  joint 
force  headquarters  might  plan  to  conduct  rapid,  decisive  operations  (RDO)  within  a  distributed, 
collaborative  information  enviromnent  (CIE)  with  coalition  partners.  The  remaining  two 
experiments  will  build  upon  the  lessons  from  MN  LOE  I  and  II  and  specifically  will  address 
knowledge-building,  command  and  control  (C2),  and  operations  in  each  subsequent  event.  In 
addition,  this  series  of  experiments  will  feed  the  concept  development  and  prototyping  paths, 
contributing  to  USJFCOM’s  transformation  strategy.  (See  Figure  1.) 
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Figure  1.  Two-Path  Architecture 


The  goal  of  MN  LOE  II,  a  process-refinement  experiment  conducted  February  10- 
28,  2003,  was  to  examine  how  to  build  a  multinational  operational  net  assessment  (ONA) 
using  a  distributed,  collaborative  information  environment.  The  experiment  also  focused  on 
information  release  issues  among  multinational  partners. 

Additionally,  it  assessed  the  viability  of  the  multinational  information  sharing  for 
allies  and  coalition  partners  (MNISACP)  concept  of  operations  developed  by  the 
USJFCOM  J6.  All  the  participants  of  MN  LOE  II — Australia,  Canada,  Germany,  the 
United  Kingdom,  and  the  United  States —  are  members  of  the  MIC,  which,  along  with 
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France,  unanimously  designated  it  as  a  Level  2  CDE  event,  led  by  the  U.S.  NATO 
CDE/SACLANT  played  the  role  of  a  separate  nation.  France,  the  sixth  MIC  member 
nation,  actively  observed  the  experiment  in  an  effort  to  learn  the  CDE  process  and  to 
prepare  for  participation  in  MN  Experiment  III.  USJFCOM  led  the  combined  effort  from 
the  U.S.  Joint  Experimentation  site  (J9),  Lakeview  Parkway,  Suffolk.  The  remaining 
players,  with  the  exception  of  SACLANT,  participated  from  their  national  experimental 
facilities.  A  memorandum  of  understanding  (MOU)  between  Australia  and  SACLANT 
covered  their  participation  in  the  experiment. 

Participating  nations  helped  to  design  and  execute  the  experiment.  The  United  States 
ONA  process  was  introduced  through  three  workshops  prior  to  MN  LOE II  execution: 

1.  September  2002:  One-day  overview  of  ONA  political,  military,  economic, 
social,  infrastructure,  and  information  (PMESII). 

2.  November  2002:  Two-day  effort  to  establish  knowledge  baseline,  including 
the  blue  force  view  of  the  red  force,  the  blue  force  view  of  itself,  the  red  force 
view  of  the  blue  force,  the  red  force  view  of  itself,  system-of-systems  analysis, 
and  node  development. 

3.  January  2003  workshop:  Conducted  over  Combined  Federated  Battle  Lab 
(CFBL)  net  for  LOE  participants  on  process  refinement  and  tools  training. 

Section  1  describes  the  purpose,  scope,  and  objectives  of  MN  LOE  II.  Section  2 
provides  an  overview  of  the  experiment  design.  MN  LOE  II  analysis  strategy,  ONA 
preparation  and  usage,  technical  command,  control,  communications,  computer,  and 
information  (C4I)  aspects,  and  the  information-sharing  construct  are  detailed  in  Section  3. 
Section  4  contains  the  findings.  A  summary  of  experimentation  lessons  learned  and 
recommendations  and  the  subsequent  analysis  phase  are  provided  in  Section  5.  The  report 
conclusion  is  detailed  in  Section  6. 

B.  SCOPE 

This  experiment  aimed  to  identify  and  assess  issues  associated  with  conducting  an 
ONA  in  a  collaborative  environment,  as  well  as  issues  regarding  information  sharing  across 
different  security  domains  within  a  coalition  environment.  The  experiment  results  will: 

□  Feed  efforts  to  develop  national  standing  joint  force  headquarters 

□  Inform  multinational  concept  development  and  experimentation  efforts,  as  well 
as  those  of  the  working  groups  of  the  Multinational  Interoperability  Council 

□  Help  to  assess  and  refine  the  coalition  information-sharing  operational  concept 
and  architecture. 

MN  LOE  II  goals  included: 

□  To  identify  issues  regarding  the  ability  of  national  headquarters  staffs  to 
conduct  distributed  ONAs 

□  To  refine  MNIS  operational  concept  and  architecture 
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□  To  refine  goals,  objectives,  and  the  “way-ahead”  for  Olympic  Vision  ’03, 
Olympic  Challenge  ’04,  and  other  transformation  events. 

C.  MN  LOE  II  OBJECTIVES  &  CRITICAL  OPERATIONAL  ISSUES 

“Request  you  determine  what  information  should  be  shared,  in 

addition  to  how,  when,  and  with  whom,  and  also  provide  comprehensive 

recommendations  on  a  definition  of  and  the  policy  for  operational 

requirements.  ” 

— From  Gen.  Richard  B.  Myers’  letter 
to  Gen.  William  F.  Keman 
(Recommendations  sent  to  the  Chairman, 

Joint  Chiefs  of  Staff,  April  15,  2002) 

The  two  objectives  of  this  experiment  stemmed  from  guidance  on  multinational 
information  sharing  in  a  letter  from  the  Chairman,  Joint  Chiefs  of  Staff,  to  the  commander 
of  Joint  Forces  Command  .  All  participating  nations  agreed  upon  objectives  and  critical 
operational  issues  (COIs). 

Objective  1:  Identify  and  assess  issues  associated  with  the  ability  of  national 
headquarters  staffs  to  conduct  a  distributed  ONA. 

Hypothesis:  //  partner  nations  conduct  ONA  through  distributed  collaboration, 
then  a  better  ONA  will  be  achieved. 

Critical  Operational  Issues:  Is  the  ONA  process  viable  in  a  coalition 
environment?  What  are  the  impacts  of  cultural  and/or  organizational  differences  on 
coalition  collaboration? 

Objective  2:  Identify  and  assess  issues  associated  with  collaboration  and 
information  sharing  across  different  security  domains. 

Hypothesis:  If  a  future  multinational  information-sharing  concept  is  employed  to 
conduct  collaboration  and  information  sharing  across  different  security  domains,  then 
shared  situational  awareness  will  be  improved. 

Critical  Operational  Issues:  Does  collaboration  with  coalition  partners  across 
security  domains  improve  the  ONA?  What  impact  does  infonnation  sharing  have  on  each 
phase  of  the  ONA  process?  Does  the  MNIS  future  concept  speed  up  the  ONA  process? 
Can  information  release  procedures  keep  the  ONA  data  current? 


3  References:  (a)  Chairman,  Joint  Chiefs  of  Staff  letter,  guidance  on  multinational  information 
sharing,  dated  January  25,  2002.  (b)  Commander,  Joint  Forces  Command  memorandum,  response 
to  Reference  (a). 
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D.  PARTICIPANTS/LOCATIONS 


Some  58  participants  played  from  dispersed  locations: 

□  United  States  7  planners/1 1  system-of-systems  analysts4 

□  Regional  (NATO)  3/6 

□  Australia  1/4 

□  Canada  3/4 

□  Germany  5/7 

□  United  Kingdom  2/8. 

One  hundred  and  two  people,  including  seven  senior  concept  developers  (SCDs)5, 
supported  the  experiment.  See  Appendix  G  for  a  detailed  account  of  experiment  demographics. 

□  United  States  55  [4  national  liaison  officers  (LNOs)] 

□  Regional  (NATO)  2 

□  Australia  13 

□  Canada  10 

□  Germany  10 

□  United  Kingdom  12. 

The  CFBL,  a  dedicated  wide-area  network,  provided  connectivity  for  the  MN 
LOE.  Three  central  points — the  United  States  players,  the  control  cell,  and  NATO — were 
located  at  USJFCOM  in  Suffolk,  Va.  The  national  headquarters  locations  included: 

□  DSTO,  Fern  Hill  Park,  Canberra,  Australia 

□  Canadian  Forces  Experimentation  Centre,  Ottawa,  Canada 

□  Center  for  Analyses  and  Studies,  Operations  Research  Division,  Ottobrunn,  Gennany 

□  DSTL  Portsdown  West  Facility,  Fareham  Hampshire,  United  Kingdom. 

These  organizations  provided  observers  and  survey  materials  for  experiment  analysis: 

□  Defence  Science  &  Technology  Organization  (DSTO),  Fem  Hill  Park,  Australia 

□  Defence  Science  &  Technology  Organization  (DSTO),  Edinburgh,  Adelaide, 
South  Australia 

□  Canadian  Forces  Experimentation  Centre  (CFEC),  Ottawa,  Canada 

□  Center  for  Analyses  and  Studies,  Operations  Research  Division,  Ottobrunn,  Gennany 

□  Defence  Science  &  Technology  Laboratory  (DSTL),  Portsdown  West,  UK 

□  U.S.  Army  Research  Laboratory 

□  NATO  SACLANT  Headquarters 

□  SPAWAR  Combatant  Command  Interoperability  Program  Office. 

4  See  Appendix  J  for  an  overview  of  system-of-systems  analysis. 

5  Senior  concept  developers  are  a  select  group  of  former  general  and  flag  officers  contracted  to 
participate  in  a  variety  of  activities  as  a  source  of  experience  and  knowledge  that  contributes  to  the 
growing  understanding  of  the  concepts  examined  in  the  experiment. 
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II.  EXPERIMENT  DESIGN 


A.  SCENARIO  &  VIGNETTES 

The  MN  LOE  II  scenario  and  its  four  vignettes  were  derived  from  the  scenario 
developed  for  the  Pinnacle  Impact  2003  and  Joint  Global  Wargame  2004  experiments. 
(See  Appendix  B.)  They  provided  a  focused  look  at  selected  aspects  of  a  developing 
situation  in  the  Pacific  Rim.  This  construct  helped  to  develop  a  multinational  ONA  that 
was  much  narrower  in  scope  than  that  of  a  full  coalition  ONA.  All  vignettes  stood  alone; 
one  did  not  grow  from  another,  and  the  information  gleaned  from  one  did  not  affect 
subsequent  vignette  execution. 

1.  Injects 

Twenty-three  products,  or  injects,  were  created  for  each  vignette — four  each  for 
Canada,  the  United  Kingdom,  Germany,  Australia,  and  the  United  States,  and  three  for  the 
regional  nation  (NATO  SACLANT  Headquarters).  USJFCOM  J9  experiment  controllers 
developed  92  injects  to  support  the  four  vignettes  from  unclassified  sources  that 
represented  national  intelligence  products,  diplomatic  and  law  enforcement  communiques, 
military  message  traffic,  and  verified  open-source  media.  Each  product  was  marked  with 
an  experiment  classification  and  an  initial  release  caveat,  as  delineated  by  the  MNIS 
matrix  (Figure  2).  All  products  were  created  to  stimulate  and  guide  multinational  ONA 
development  and  MNIS  processes,  given  the  limited  time  available  for  the  participants  to 
assimilate  and  enter  the  data  during  each  vignette. 

2.  Information-Sharing  Matrix 

The  MNIS  matrix  (Table  1)  comprised  six  domains:  multilateral,  trilateral, 
bilateral  1,  bilateral  2,  coalition,  and  private.  Each  represented  differing  relationships 
among  the  participating  countries.  Injects  were  developed  for  each  country  in  a  specific 
domain  to  stimulate  the  participants  to  delete,  add,  or  modify  an  effect,  node,  action,  or 
resource  (ENAR)  record  in  the  ONA  database.  Depending  on  the  product,  the  action 
would  be  accomplished  by  the  multinational  ONA  system-of-systems  analyst,  who  would 
update  the  nodes  in  the  ONA  database,  or  by  the  planners,  who  would  update  the  effects, 
actions,  or  resource  listings  in  tandem  with  the  analysts.  For  more  information  about  the 
MNIS  concept/process,  see  Sections  II  E  and  III  E. 
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Table  1.  MN  LOE II MNIS  Matrix 


B.  ANALYSIS 

Developed  as  part  of  the  design  process,  an  experiment  analysis  plan  covered: 

□  Experiment  design  and  COIs 

□  Collection  of  quantitative  and  qualitative  data  to  support  assessment  of  the 
experiment  COIs 

□  Data  collection  methods 

□  Assessment  strategy 

□  Analysis  methods. 

Participating  nations  played  an  active  role  in  the  design  and  execution  of  the 
experiment  analysis  plan.  As  a  result,  they  helped  to  identify  national  objectives,  issues, 
metrics,  and  associated  data  collection  requirements. 

The  experiment  analysis  plan  included  the  four  data  collection  methods  used: 

□  Analyst  participation  as  control  cell  members 

□  SCD  interactions  and  discussions 

□  Players’  questionnaires 

□  Observations. 

For  information  about  the  data  collection  and  assessment  methodology,  see  Section  III. 


C.  MN  LOE  II  ARCHITECTURE 

MN  LOE  II  consisted  of  three,  five-day  periods  during  which  the  participants 
focused  on  different  aspects  of  the  multinational  ONA  and  deliberate  planning  process. 
Week  0  (February  10-14,  2003)  was  devoted  mainly  to  tools  refresher  (Groove,  ONA 
graphical  layout)  and  process  training  for  all  participants.  Weeks  1  and  2  were  used  to 
examine  the  building  of  the  ONA.  Week  1  (February  17-21)  examined  the  “as  is”  MNIS 
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environment  during  pre-coalition  multinational  ONA  development,  while  Week  2 
(February  24-28)  considered  the  future  MNIS  environment. 

1.  Distributed  Collaboration  Instrument 

MN  LOE  II  used  a  commercially  available  software  package  known  as  “Groove” 
that  enables  distributed  collaboration.  This  technology  was  used  during  the  preplanning 
phase  to  allow  collaboration  during  scenario  development  and  in  preparation  for 
experiment  analysis.  Groove  included  several  workspaces,  a  graphical  layout  (GL)  tool 
that  allows  the  user  to  visually  access  the  linked  portions  of  the  ONA  database,  text  chat  in 
all  spaces,  audio  communications,  and  private  e-mail  chat  capability. 

2.  Pre-execution 

The  months  preceding  the  experiment  included  several  planning  meetings, 
workshops,  development,  and  refinement  of  the  vignettes  and  injects.  Early  in  the 
process,  the  multinational  partners  agreed  on  the  four  basic  vignette  focus  areas  and 
discussed  various  aspects  of  the  MNIS  process.  Later,  during  the  course  of  three  ONA 
workshops,  the  participants  refined  various  aspects  of  the  ONA  and  effects,  actions,  and 
resource  listings.  In  three  ONA  workshops,  the  U.S.  ONA  process  served  as  a  model.  A 
compact  disc  (CD)  containing  the  scenario  and  vignette  information,  injects,  concept 
papers,  briefings,  and  the  experiment  overview  briefing  was  distributed  to  the  J9  “trusted 
agents”  who  deployed  to  the  participating  countries  during  the  execution  phase.  Each  of 
these  trusted  agents  would  extract  his  country’s  injects  and  post  them  into  the  respective 
“private”  country  domain  within  the  Groove  shared  space.  Because  of  the  lead  time 
needed  to  create  the  injects  and  to  produce  and  distribute  the  CD,  the  J9  experiment 
controllers  provided  early  lists  of  proposed  effects,  actions,  and  resources  for  use  during 
the  inject  development  phase,  rather  than  the  final  lists  from  the  third  workshop. 

3.  Week  0 

The  first  week  reinforced  the  basic  concepts  of  effects-based  operations  (EBO),  ONA, 
and  MNIS,  while  simultaneously  providing  additional  tools  refresher  training  to  all 
experiment  participants.  Reinforcement  of  the  U.S.  ONA  process  began  with  reviews  of  the 
effects,  actions,  and  resources  products  that  were  developed  during  the  three  workshops  and 
then  rolled  into  a  controlled  walk-through  of  the  entire  process,  using  the  collaborative  tools. 

4.  Weeks  1  &  2 

Experiment  play  started  with  a  short  scenario  briefing  and  an  introduction  of  the 
vignette  to  participants.  Both  were  posted  in  Groove  for  their  review,  along  with 
supporting  documents.  Vignette  descriptions  remained  posted  after  their  completion. 
While  the  vignettes  were  not  linked,  the  same  data  originally  planted  within  the  ONA 
database  was  used  for  each  of  them. 
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Each  country’s  trusted  agent  extracted  the  four  injects  for  each  country,  as  well  as 
three  for  the  regional  player,  from  the  MN  LOE  II  CD  and  posted  them  in  that  country’s 
private  Groove  shared  space  for  review  and  action,  i.e.,  withholding  the  product  completely  or 
posting  it  to  other  shared  spaces,  in  a  sanitized  fonn  or  in  its  entirety.  If  further  dissemination 
of  the  information  was  deemed  necessary,  the  product  would  be  reviewed  through  the 
originator’s  disclosure  process,  and  if  approved,  posted  in  other  shared  spaces  on  Groove  with 
an  updated  classification  release  caveat.  Upon  receipt  of  the  new  product,  the  plans  and 
system-of-systems  analysis  teams  would  use  the  Groove  collaborative  tool  to  work  through 
the  effects,  actions,  and  resources  process.  This  process  was  repeated  for  each  vignette. 

D.  VIGNETTE  SCHEDULE 

Each  of  four  vignettes  focused  on  a  different  aspect  of  the  developing  regional 
situation,  while  stressing  multinational  ONA  development  and  MNIS.  The  participants 
examined  each  vignette  during  a  two-day  period,  executing  two  vignettes  per  week.  The 
remaining  day  included  after- action  reviews  and  national  “hot  wash”  discussions.  During 
Week  1,  the  first  two  vignettes  concentrated  on  two  focus  countries  with  separate  and 
limited  connections  whose  militaries  operated  independently  of  each  other.  During  Week 
2,  the  third  and  fourth  vignettes  used  the  premise  that  both  focus  countries  may  federate 
and  that,  therefore,  the  multinational  ONA  and  planning  process  should  examine  the  effect 
of  a  combined  regional  power  rather  than  of  two  independent  ones. 

E.  INFORMATION  SHARING 

Discussion  about  the 
execution  of  the  information¬ 
sharing  portion  of  MN  LOE  II 
demands  review  of  the  experiment 
design  used  to  examine  the  U.S. 

MNIS  operational  concept 
(OPCON).  The  MNIS  OPCON 
defines  a  future  vision  where 
information  is  protected  at  its 
source  and  is  shared  with  coalition 
partners  across  a  single  infor¬ 
mation  sharing  enviromnent.  In 
addition,  all  coalition  partners 
would  not  have  access  to  all  infor¬ 
mation,  and  classification  may  be 
used  to  control  access.  This  vision 
applied  to  a  wide  range  of  activi¬ 
ties,  from  prehostilities  through 
postcombat  planning  and  exe¬ 
cution.  Access  was  based  on  controls  enforced  at  information  destinations. 


Figure  2.  Current  Information-Sharing  Design 
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Is  the  future  information¬ 
sharing  vision  better  than  the 
current  system?  Current  infor¬ 
mation  sharing  includes  multiple 
security  domains,  each  supporting 
bi-,  tri-,  and  multilateral  sharing 
agreements.  (See  Figure  2.)  To 
release  information  outside  the 
sharing  agreement,  the  informa¬ 
tion  originator  followed  national 
disclosure  procedures.  If  the  in¬ 
formation  was  developed  collabor- 
atively,  all  parties  first  must  agree 
to  further  disclosure  before  the 
disclosure  process  started. 


Originally,  future  informa¬ 
tion  sharing  was  seen  as  a  single 
security  domain  where  all  information  would  be  released.  Rather  than  retrieving 
information  from  separate  security  domains,  participants  would  view  all  information 
released  to  them  in  the  coalition  security  domain.  (See  Figure  3.)  A  participant’s 
nationality  and  his  nation’s  participation  in  the  relevant  information- sharing  agreement 
would  determine  access.  However,  national,  bi-,  tri-,  and  multilateral  domains  remained 
within  the  single  security  domain,  and  national  information  was  not  placed  within  the 
coalition  security  domain. 

For  both  current  and  future  information-sharing  treatments,  Groove  shared  spaces 
were  used  in  conjunction  with  the  ONA  database  security  features  to  establish  security 
domains.  The  LOE  technical  team  invited  participants  to  join  a  shared  space  and 
controlled  access  to  shared  spaces.  All  participants  had  access  to  the  coalition-shared 
space,  spaces  representing  the  multinational  and  bilateral  agreements,  and  to  their 
respective  national  shared  space. 

The  technical  team  also  set  up  the  ONA  database  with  fields  for  disclosure  to 
sharing  agreements  for  each  node.  Reference  files  linked  to  a  node  also  had 
accompanying  disclosure  tags.  When  the  current  information-sharing  treatment  was  in 
effect,  participants  used  their  Groove  identities  to  enter  the  shared  space  to  view  ONA 
database  node  records  for  that  sharing  agreement.  Participants  could  view  and  retain 
information  not  released  by  their  nation  by  saving  infonnation  within  their  national  shared 
space.  Since  only  node  information  was  tagged,  participants  could  not  go  to  a  single 
shared  space  to  view  all  information  released  to  their  nation.  According  to  the  experiment 
plan,  when  the  future  information- sharing  treatment  was  in  effect,  participants  could  view 
all  information  released  to  their  nation  in  the  coalition  shared  space,  including  their 
respective  national  information  not  released  to  others. 


Figure  3.  Future  Information-Sharing  Design 
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The  assumption  a  priori  was  that  the  null  hypothesis — no  difference  exists 
between  the  information-sharing  treatments — would  be  rejected.  Shared  awareness 
should  increase  when  all  information  that  is  released  to  participants  is  available  in  a 
collaborative  shared  space. 

Measuring  shared  awareness  during  the  infonnation-sharing  treatments  began  as  the 
control  team  distributed  disclosures  to  participants  at  the  beginning  of  each  vignette.  Each 
treatment  was  exercised  during  two  sequential  16-hour  vignettes  perfonned  in  eight-hour 
segments.  Participants  answered  “probe”  questions  periodically,  and  the  responses  were  used 
to  establish  situational  awareness.  At  the  end  of  each  experiment  segment,  participants  also 
completed  questionnaires  and  provided  comments  about  the  infonnation-sharing  treatments. 

The  LOE  exercised  the  Coalition  Information-Sharing  Policy  and  Procedures  Panel 
(CISP3)  as  an  adjunct.  The  CISP3  concept  evolved  from  the  Command-and-Control 
Interoperability  Board  (CCIB)  used  in  the  U.S.  Pacific  Command.  The  capability  to 
coordinate  common  requirements  for  sharing  information  was  likened  to  the  CCIB  process 
to  establish  interoperability  to  the  greatest  extent  possible.  As  the  MNIS  OPCON 
developed,  the  CISP3  concept  emerged  as  a  capability  that  could  enhance  efficiency  for 
the  coalition  as  a  cohesive  force  from  the  start.  The  panel  would  establish  a  common  set 
of  information- sharing  policies  and  procedures  to  support  the  coalition  commander’s 
intent,  bringing  with  it  national  limitations  and  sensitivities. 

The  CISP3  establishes  and  maintains  policy  and  procedures  that  permit  the 
creation  of  coalition  infonnation-sharing  environments.  The  LOE  could  not  examine  all 
aspects  of  the  panel.  In  accordance  with  the  experiment  design,  a  CISP3  member  from 
each  nation  helped  to  develop  information  exchange  requirements  (IERs),  which  are 
needed  to  build  a  coalition  knowledge  base.  Most  participants  agreed  that  the  IERs  were 
too  cumbersome  and  that  the  role  and  functions  of  the  CISP3  were  poorly  defined.  To 
accelerate  information  release  during  Week  2,  the  CISP3  identified  categories  of 
releasable  information  without  following  national  disclosure  procedures.  Each  national 
panel  developed  a  list  based  on  Annex  B  of  the  MNIS  OPCON  document.  Analysts  and 
planners  then  used  the  lists  to  determine  the  information  to  be  released  without  going 
through  formal  disclosure  processes. 
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III.  EXPERIMENT  EXECUTION 


Daily  experiment  execution  was  governed  by  the  schedule  presented  at  the 
beginning  of  each  day  by  the  chief  controller.  Constructed  as  part  of  the  experiment  design, 
this  schedule  was  fine-tuned  for  the  upcoming  day  using  observations  and  suggestions  from: 
1)  senior  concept  developers,  2)  chief  controller’s  end-of-day  hot  wash  with  national  lead 
controllers  and  MN  LOE  II  functional  leads,  3)  the  plans  and  system-of-systems  analysis 
(SoSA)  team  leads,  4)  the  technical  director,  and  5)  the  experiment  director.  Each 
experiment  day  began  with  a  Groove  oral  and  text-chat  national  check-in  with  the  chief 
controller.  He  reviewed  the  day’s  experiment  objectives,  the  daily  focus  for  collaboration, 
the  status  of  the  systems,  issues  to  be  resolved,  and  daily  surveys  to  be  completed.  Plans 
and  SoSA  team  leads  then  began  the  daily  session.  Each  day  started  and  ended  on  time. 

A.  EXPERIMENT  WEEK  1  AND  WEEK  2 

Week  0,  the  final  training  session,  focused  mainly  on  using  the  collaborative  tool, 
the  database,  various  other  tools,  the  ONA  concept,  and  the  experiment  scenario.  Week  1 
introduced  the  experiment  audience  to  the  multinational  ONA  development  process,  in 
which  the  ONA  process  of  knowledge  discovery  from  the  vignette  injects  was  combined 
with  the  development  of  a  multinational  ONA.  Week  2  broadened  the  systems-of-systems 
approach  to  include  more  effect,  node,  and  action  options  leading  to  the  development  of  a 
shared  knowledge  base. 

1.  Experiment  Week  1 

Vignette  1  presented  the  scenario  framework  of  the  focus  nations’  improved 
military  capabilities.  Plans  and  SoSA  chose  to  work  separately,  and  considerable  time 
was  spent  redefining  effects  developed  during  Week  0  to  match  the  scenario/vignette. 
They  also  identified  applicable  nodes.  Experiment  control  monitored  some  analyst- to- 
analyst  private  chats,  public  text  chat,  and  intennittent  Groove  audio  to  observe  player 
actions.  Vignette  1  promoted  an  understanding  of  the  relationship  of  new  information  to  a 
framework  of  predetermined  effects.  Vignette  2  combined  player  understanding  of 
applicable  nodes  with  the  refined  effects  within  a  scenario  that  now  included  political  and 
economic  considerations.  The  throughput  of  vignette  information  improved  as  analyst-to- 
analyst  confidence  developed.  Communications  anomalies  in  the  Groove  system, 
especially  in  the  audio,  necessitated  more  “national  only”  node  and  effect  development, 
resulting  in  some  redundancy  among  nations.  This  imposed  an  unrealistic  constraint  upon 
the  numbers  of  effects  and  nodes  to  be  addressed  by  the  experiment  audience.  Vignette  2 
promoted  a  dialogue  that  presented  national  perspectives;  differing  interpretations  of 
effects,  actions,  and  resources  in  the  context  of  the  scenario;  and  a  greater  willingness  to 
collaborate  in  functional  areas. 
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2.  Experiment  Week  2 

In  Vignette  3,  continued  refinement  of  the  multinational  ONA  process  defined  in 
Week  1  combined  the  improved  effects  with  selected  nodes,  and  then  applied  potential 
actions  and  resources  within  the  scenario  framework.  The  control  cell  monitored  and 
orchestrated  game  progress  through  the  network  of  trusted  agents,  since  most  work  was 
conducted  in  private  chat  or  face-to-face.  Vignette  information  moved  more  quickly  to  a 
broader  audience  as  the  players  recognized  that  the  information  affected  multiple  nodes 
and  effects,  thus  requiring  different  actions. 

At  the  request  of  the  U.S.  SoSA  lead,  the  chief  controller  introduced  a  daily  plan 
that  kept  the  players  on  track  through  the  combined  interactive  deliberations.  Control 
monitored  activity  through  text  chat,  private  chat,  Groove  audio,  and  audio  speakerphones. 
SoSA  role  players  stimulated  discussion.  Vignette  3  emphasized  that  multinational  ONA 
was  not  part  of  the  intelligence  process,  separated  some  national  players  from  a  "pull 
mentality,"  and  realized  an  SCD  desire  for  introduction  of  national  disagreements. 

Vignette  4  capitalized  upon  the  improved  interactions,  the  daily  schedule,  and  the 
player-developed  ENAR  spreadsheets  from  Vignette  3  within  a  framework  of  focus 
nations’  political  and  military  cohesion.  Productivity  increased  as  SoSAs  were  grouped 
by  multinational  PMESII  disciplines  and  presented  their  nominations  to  the  multinational 
plans/SoSA  forum.  Serving  as  a  backdrop  for  functional  refinement,  Vignette  4  examined 
the  DIME  wargaming  process  for  the  first  time  since  the  LOE  workshops.  Introduction  of 
DIME,  as  well  as  the  visual  presentation  of  ENAR  linkages  using  the  ONA  graphical 
layout  tool,  stressed  the  complexity  of  multinational  ONA  development  to  the 
international  players  for  the  first  time.  Vignette  4  promoted  the  systems-of-systems 
approach,  the  need  to  introduce  information  rapidly  into  the  intellectual  analyst  forum,  and 
the  acknowledgement  of  a  changing  environment  within  a  seemingly  hierarchically 
structured  multinational  ONA  development  process. 

B.  EXPERIMENT  CONTROL 

Because  the  experiment  was  distributed,  MN  LOE  II  control  personnel  in  each  of  the 
participating  countries  communicated  through  the  Groove  collaborative  tool  and  the 
telephone.  Overall  experiment  control  focused  on  a  small  cadre  of  individuals  located  in  the 
J9  facility  in  Suffolk,  Virginia.  Each  participating  nation’s  lead  controller  conducted  the 
experiment  within  that  country.  Liaison  officers  (LNOs)  at  the  J9  facility  and  U.S.  LNOs  in 
each  country  provided  a  national  cultural  bridge.  Although  not  part  of  the  fonnal  experiment 
control  network,  the  LNOs  provided  an  additional  conduit  and  on-scene  concept/experiment 
expertise.  See  Appendix  C  for  a  more  infonnation  on  the  experiment  control  process. 

1.  LOE  Management 

The  Groove  tool  was  the  primary  means  of  collaboration  among  the  participants 
and  among  control  personnel.  Three  shared  spaces  within  Groove  were  set  aside  for 
experiment  control  functions:  control  cell,  white  cell,  and  auditorium.  Most  of  the 
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experiment  audience  participants  could  not  view  each  space.  Private  chats  and  messages 
through  the  Groove  system  also  allowed  direct  communication  among  all  players  for 
control  and  experiment  participation.  In  addition,  Internet  Protocol  phones  were  used  to 
control  private  and  conference  call  modes. 

a)  Control  Cell  Shared  Space 

This  was  the  primary  control  cell  communication  avenue.  Access  was  limited  to 
national  control  cell  leads,  liaison  officers,  and  functional  leads. 

b)  White  Cell  Shared  Space 

Intended  to  provide  a  limited-access  space  that  would  allow  discussions  among 
national  white  cell  personnel,  this  shared  space  was  not  used  in  any  meaningful  manner 
during  the  LOE. 

c)  Auditorium  Shared  Space 

This  shared  space  was  intended  to  grant  access  to  all  participants  and  control 
personnel  for  mass  training  sessions  (Week  0)  and  experiment-related  briefings.  In 
practice,  when  a  significant  number  of  participants  entered  the  space,  the  audio  became 
unintelligible.  As  a  resolution,  access  to  the  auditorium  shared  space  was  deleted  from 
most  computers;  only  one  per  work  area  could  enter  this  Groove  space.  This  single 
computer  was  connected  to  a  projector  and  external  speakers,  thus  allowing  a  relatively 
large  audience  to  view  and  hear  the  presentation.  Despite  this  improvement,  the  poor 
quality  of  the  audio  using  Groove  had  a  detrimental  impact  upon  the  experiment. 

d)  Private  Text/Audio  Chats  and  Messages 

The  Groove  system  allowed  the  participants  to  conduct  private  text  or  audio  chats 
among  themselves.  Additionally,  private  messages  were  widely  used  during  the 
experiment  by  both  participants  and  control  personnel.  These  capabilities  enabled  the 
control  cell  to  resolve  individual  concerns  privately. 

e)  Internet  Protocol  Phones 

The  dial-up  Internet  Protocol  phones  were  used  to  back  up  the  auditorium’s  audio. 
Extreme  problems  with  poor  quality  Groove  audio  could  be  circumvented  using  Internet 
Protocol  phone  conference  calls  with  a  visual  presentation  through  the  auditorium 
projection  system  or  individual  computers. 

C.  SYSTEM  PERFORMANCE/TECHNICAL  SUMMARY 

All  multinational  technical  support  to  MN  LOE  II  was  excellent,  and  Groove 
required  much  technical  support  to  continue  to  run  smoothly.  With  the  exception  of  audio 
problems,  the  network,  hardware,  and  software  facilitated  experiment  execution.  The 
ONA  database  provided  excellent  support  for  the  event,  but  could  be  faster  and  more  user 
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friendly.  The  ONA  graphical  layout  and  topographical  map  tools  were  not  used  often  due 
to  time  constraints;  their  speed,  flexibility,  and  user-friendliness  also  could  be  improved. 
System  performance  of  the  CFBLNet  and  of  the  J9  setup  is  detailed  here. 

1.  Combined  Federated  Battle  Laboratories  Network  (CFBLNet) 

Collaborative  planning  for  MN  LOE  II  was  conducted  using  the  classified 
Combined  Federated  Battle  Lab  network  (CFBLNet)6,  which  allowed  the  MN  LOE  II  to 
be  conducted  in  a  distributed  global  manner,  with  nations  participating  from  facilities  in 
their  own  country.  Use  of  the  CFBLNet  and  available  national  facilities  netted 
considerable  savings  in  travel  costs.  In  addition,  the  network  was  available  in  all  the 
nations  to  support  the  experiment,  which  facilitated  the  participation  of  national  staff. 

a)  Dynamic  Environment 

Overall,  CFBLNet  management  succeeded  in  resolving  conflicts  between  MN 
LOE  II  and  other  experimental  activities.  Conducting  activities  on  CFBLNet  still  proved 
challenging  due  to  the  sheer  number  of  organizations  with  a  stake  in  its  maintenance. 
Two  maintenance  actions  interfered  with  event  execution:  Australia  effectively  lost  three 
days’  participation  during  Week  0  due  to  an  uncoordinated  network  upgrade,  and  another 
maintenance  action  was  stopped  before  it  affected  the  event.  The  most  challenging  part  of 
using  this  network  is  communicating  effectively  with  all  of  the  key  players  in  its 
operation.  Participant  nations  must  work  closely  and  early  with  CFBLNet  management 
and  other  agencies  to  ensure  that  all  events  and  maintenance  activities  do  not  conflict  with 
scheduled  experiments. 

b)  Network  Monitoring 

Network  monitoring  and  diagnostic  tools,  such  as  Qcheck,  MRTG,  and  HP 
Openview,  proved  invaluable  in  tracking  network  anomalies  and  inefficiencies  during  the 


6  The  CFBL  initiative  is  a  consortium  developed  among  the  U.S.,  NATO,  and  nations  involved 
with  the  Combined  Communications-Electronics  Board.  “The  CFBL  net  is  a  longer-term 
combined  research,  development,  test,  and  evaluation  network.  It  is  not  intended  to  be  a 
combatant  network  or  the  architecture  for  a  future  network.  This  network  is  projected  to  remain 
in  operation  to  conduct  coalition  C4I  experiments  and  [to]  provide  possible  parallel  use  in 
[combatant  commander]  coalition  exercises.  Creation  of  the  CFBL  net  leverages  joint  warrior 
interoperability  demonstration  resources,  existing  U.S.  federated  battle  laboratories  assets,  and 
coalition  battle  laboratories/test  beds.  As  such,  it  will  not  be  a  solely  U.S. -owned/operated 
network,  but  a  combined  network,  with  the  members  having  equal  say  in  its  utilization  and 
management.  NATO  nations  participating  in  bilateral  or  multilateral  CFBL  net  project 
arrangements  are  responsible  for  funding  their  portion  of  the  effort,  and  there  is  no  requirement 
for  monies  to  be  exchanged  among  participants.  The  NATO  point  of  entry  for  [the]  CFBL  net  is 
NC3A  The  Hague.”  [Joint  Warrior  Interoperability  Demonstration  2002  Web  site, 
http://www.iwid.is.mil/html/cfblnet.htmLl  CFBL  net  was  chosen  for  MN  LOE  II  because  it  is 
the  only  established  and  accredited  wide-area  network  available  for  experimentation  with 
multinational  partners. 
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work-up  spirals  and  execution.  Seven  such  significant  WAN  anomalies  were  corrected, 
including  three  Domain  Name  Server  errors  in  routers,  two  half-duplex/full-duplex  setting 
mismatches,  high  latency  routes,  Open  Shortest  Path  First  routing  anomalies,  and  more. 
Although  CFBLNet  personnel  provided  network  monitoring  support,  local  monitoring  at 
participant  sites  proved  to  be  invaluable,  often  helping  to  remediate  network  anomalies 
quickly.  Future  events  on  CFBLNet  should  continue  to  provide  participant  network 
monitoring,  as  well  as  to  develop  closer  ties  with  the  CFBLNet  management. 

c)  JFCOM  Support 

JBC,  J6,  and  J9  must  collaborate  to  detennine  the  JFCOM  entity  to  support  the 
CFBL  net  point  of  participation  for  future  multinational  events. 

2.  J9  MN  LOE  Network 

a)  Account  and  Permissions  Management 

Technical  requirements  for  participant  setup  actions  must  be  linked  directly  to  the 
manning  document.  To  avoid  the  need  for  technicians  to  update  technical  accounts  and 
user  pennissions  as  the  document  changes,  it  could  be  integrated  directly  into  the 
database,  allowing  automatic  updates  upon  command. 

b)  ONA  Database 

Exposure  of  the  ONA  database  to  the  multinational  community  led  to  these 
insights  that  may  improve  its  future  operation: 

□  The  user  interface  should  be  faster  to  allow  direct  entry  of  data  instead  of  using 
Excel  or  other  tools  for  intermediate  data  capture. 

□  Improved  user-friendliness  of  the  interface  would  greatly  enhance  the 

productivity  of  the  average  user. 

□  To  support  real-world  coalition  operations,  the  database  should  be  migrated  to 
an  integrated  multidomain,  multilevel,  secure  architecture  that  allows  each  user 
to  see  a  fused  picture  of  the  ONA  across  all  classifications  and  releasability 
levels  to  which  they  have  access. 

As  part  of  this  migration  to  a  multidomain  ONA,  an  Extensible  Markup  Language 
export  feature  should  be  added  to  facilitate  data  exchange  between  distributed  coalition 
participant  national  databases.  Each  of  these  databases  would  be  owned  and  operated  by 
the  respective  participant  nation,  and  they  would  control  the  release  of  information  to 
other  participants  as  directed  by  national  infonnation-sharing  policies.  The  mechanism 
for  releasing  data  would  be  policy-independent.  Each  nation  would  have  the  option  to  run 
a  multidomain  ONA  database,  and  they  would  “plug  and  play”  in  a  coalition  environment. 
Each  could  replicate  data  to  other  coalition  members,  using  the  XML  export  feature. 
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c)  ON  A  Graphical  Layout  (ON A  GL) 


The  ONA  graphical  layout  tool  provides  a  visual  interpretation  of  the  relationships 
of  data  within  the  ONA  database.  Although  available  during  MN  LOE  II,  this  tool  was 
not  essential,  since  the  event  primarily  involved  populating  the  ONA  database;  the  tool 
will  be  more  valuable  during  operational  planning.  After  minimum  exposure,  feedback 
indicates  that  the  tool  must  be  more  closely  integrated  with  the  Web  application  and 
mapping  tools,  including  initiating  database  updates  from  this  interface,  and  that  it  needs 
more  flexible  filtering  options.  In  addition,  the  default  view  should  be  user  selectable. 

d)  ONA  Node  Topographic  Map 

The  ONA  node  topographic  map  tool  similarly  was  not  used  much  in  this  event.  It 
provides  a  detailed  visual  perspective  of  the  topographical  relationships  among  nodes  that 
will  support  operational  planning.  It  will  be  more  valuable  in  the  next  experiment  in  this 
series.  Many  of  the  same  recommendations  for  ONA  GL  apply. 

e)  Web  Site 

No  technical  support  Web  site  was  available  during  MN  LOE  II  to  share 
configuration  information,  software,  and  software  fixes/patches.  Such  a  site  may  have 
alleviated  some  of  the  technical  issues  encountered,  and  it  should  be  implemented  for 
future  events. 


f)  Groove/Network  Bandwidth 

Groove  provides  capabilities  not  currently  available  in  any  other  software,  and  if 
managed  correctly,  it  can  be  an  enhancement  to  any  collaboration  suite.  Groove  was  selected 
for  this  event  because  it  provides  encrypted  voice  and  text  chat,  and  since  it  is  a  P2P 
application,  it  does  not  rely  on  servers  and  is  inherently  more  survivable.  Each  client 
maintains  a  copy  of  the  infonnation  on  its  hard  drive.  However,  Groove  was  used  outside  its 
capability  to  support  this  experiment.  It  is  not  designed  for  use  in  large  collaboration 
environments;  its  capacity  is  about  20  participants.  Specific  difficulties  encountered  include: 

(1)  Audio  Quality,  Reliability,  Scalability 

The  cause  of  audio  problems  cannot  be  attributed  specifically  to  Groove  or  to  the 
network  bandwidth  allocated.  Groove  audio  quality  was  good  in  shared  spaces  of  10 
members  or  less.  It  degraded  gradually  in  shared  spaces  of  10-50  members  and  was 
totally  unusable  at  60  users  in  a  shared  space.  Its  effectiveness  was  nullified  with  as  few 
as  1 5  users  when  significant  bandwidth  was  consumed  during  other  simultaneous  network 
activities.  During  a  network  stress  test,  it  cut  out  altogether.  Due  to  the  P2P  nature  of 
Groove  communications,  these  numbers  are  not  scalable  with  additional  or  upgraded 
equipment  or  network  bandwidth.  The  baseline  system  employed  Dual  Athlon  1.5  GHz 
processors  with  1Gb  RAM  on  a  100Mbps  fiber  network.  The  minimum  bandwidth 
available  to  distributed  locations  was  1.5  Mbps. 
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(2)  Nature  of  P2P 

The  bandwidth-intense  nature  of  P2P  communications  in  a  network  this  large 
mandated  that  the  network  be  pristine.  Groove’s  performance  proved  to  be  an  indicator  of 
overall  network  health.  If  a  node’s  latency  was  high  for  any  reason,  performance  degrades 
were  immediately  apparent.  In  MN  LOE  II,  the  CFBL  net  was  not  used  for  any  other 
experimentation  efforts  and  so  was  pristine.  Within  its  design  envelope,  Groove  operates 
in  noisy,  disconnected,  and  dynamic  networks.  Problems  most  likely  occurred  because  the 
experiment  tried  to  operate  Groove  outside  of  its  design  envelope. 

(3)  Groove  Management  Servers 

For  the  purpose  of  the  experiment,  a  Groove  relay  server  was  operated  at  the 
Australian  site  of  DSTO  Fem  Hill  Park  in  Canberra.  This  was  integrated  into  the  other 
Groove  management  servers  maintained  at  J9.  All  other  participants  relied  upon  the  U.S.  J9 
Groove  management  and  relay  server  in  a  star  topology.  During  part  of  the  experiment,  the 
U.S.  LNO  Groove  client  worked  with  the  J9  relay  server.  When  switched  to  the  local  relay 
server,  synchronization,  audio,  and  propagation  of  deltas — i.e.,  changes  in  the  Groove 
spaces — improved.  Thus,  in  any  future  experiment  or  deployment  that  uses  Groove,  the 
design  of  the  Groove  architecture  and  supporting  management  services  should  be  considered. 

(4)  Fetch  Loops 

The  condition  known  as  “fetch  loops”  is  an  anomaly  within  Groove  that 
substantially  slows  down  the  network  performance.  This  phenomenon  was  encountered 
twice  during  execution  and  three  times  during  testing.  In  the  worst  case,  a  distributed  user 
placed  ~4.5  Mb  of  files  into  a  space  shared  by  more  than  100  users.  This  immediately 
stuffed  the  El  service  line  for  two  to  12  hours,  depending  upon  location  and  position  in 
the  P2P  distribution  queue.  When  Groove  traffic  stuffs  a  network  “pipe,”  the  transmission 
error  rate  increases  considerably.  Groove  ensures  that  all  information  gets  synchronized 
and  sends  a  retrieval  message  when  it  detects  dropped  packets.  These  messages  stack  up 
into  the  already  jammed  pipe  and  generate  more  dropped  packets,  which  generate  more 
retrieval  messages,  and  so  on. 

(5)  Synchronized  Not 

On  several  occasions,  user  shared  spaces  indicated  they  were  synchronized  when,  in 
fact,  they  were  not,  yet  no  error  was  reported.  As  a  result,  the  user  mistakenly  believed  that 
data  was  current.  Upon  discovery,  the  condition  cleared  after  deleting  the  shared  space  from 
the  offending  machine  and  re-inviting  the  participant  to  the  shared  space.  This  condition  is 
unacceptable.  If  synchronization  is  lost,  faults  must  be  reported  reliably.  The  absence  of 
fault  reporting  for  this  condition  could  have  grave  consequences  in  a  military  enviromnent. 

(6)  Chattiness  Among  Servers 

Groove’s  continual  need  to  communicate  between  the  management  and  database 
servers  was  generating  so  much  “chatter”  that  eventually  they  were  merged  onto  one  Quad 
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server  to  improve  system  performance.  The  same  should  be  done  for  any  future  events 
that  use  Groove  in  the  database  interface. 

(7)  Difficult  To  Run  on  a  Closed  Network 

Groove  Version  2.1  uses  the  World  Wide  Web  to  link  back  to  Groove.net  servers 
for  updates.  When  set  up  in  a  closed  network,  it  generated  thousands  of  attempts  to  access 
the  Web.  This  functionality  had  to  be  disabled  and  would  make  Groove  difficult  to 
accredit  on  operational  systems  without  a  thorough  evaluation  of  the  information  that  is 
continually  being  transferred  via  the  umbilical  to  Groove.net. 

(8)  Accrual  of  Data  in  the  Error  Logs 

Approximately  3Gb  of  data  accumulated  in  the  Groove  relay  error  logs  daily.  The 
relay  server  was  set  to  purge  automatically  after  10  days,  but  didn’t.  Attempts  to  initiate 
the  purge  using  the  Purge  button  failed,  and  therefore,  files  had  to  be  manually  deleted. 
The  cause  of  this  anomaly  is  undetennined;  it  may  be  related  to  Defense  Infonnation 
Systems  Agency  security  lockdowns.  Periodically,  error  logs  must  be  deleted  manually  to 
prevent  the  Groove  relay  server  from  crashing. 

(9)  No  Support  for  Roaming  Profiles 

To  move  users  between  machines  is  a  manual  process.  Groove  does  not  support 
roaming  profiles.  To  compensate,  Groove  accounts  should  be  exported/saved  to  a  shared 
drive  to  support  account  recovery  in  the  event  of  a  PC  failure. 

(10)  High-Maintenance  Application 

Peer-to-peer  resynchronization  times  required  technicians  to  start  up  workstations 
two  to  three  hours  before  execution  daily.  This  resulted  in  days  that  averaged  90  minutes 
longer  than  with  similar  client-server  collaboration  tools.  Groove  performed  better  when 
it  communicated  continuously.  During  MN  LOE  II,  many  systems  had  to  be  powered 
down  to  remove  and  store  hard  drives  daily.  This  contributed  to  synchronization  issues 
during  system  power  up.  When  Groove  detects  changes,  it  does  not  immediately  push  the 
changes  out  to  all  the  other  potential  users.  Built  within  the  application  are  algorithms  that 
are  invoked  to  balance  the  distribution  of  changes  based  upon  knowledge  of  the  network, 
bandwidth,  availability  of  other  clients,  and  relay  servers. 


D.  ONA  PREPARATION  AND  USAGE 

Development  of  the  ONA  database  was  a  shared  responsibility  of  participants  from 
all  multinational  partners.  As  the  ONA  concept  proponent,  the  U.S.  led  this  effort  and 
trained  and  mentored  other  multinational  participants  throughout  the  development  process. 
See  Appendix  E  for  a  detailed  listing  of  activities  that  occurred  prior  to  MN  LOE  II 
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execution.  In  order  to  develop  a  common  understanding  of  the  ONA  concept,  the  concept 
lead  presented  a  briefing  during  the  May  2002  concept  development  conference. 

By  the  end  of  June  2002,  a  scenario  had  been  selected  for  MN  LOE  II,  and  work 
began  on  a  read-ahead  package  that  would  provide  multinational  participants  with  the 
understanding  or  foundation  to  develop  a  knowledge  base  for  the  scenario  countries.  By 
mid-July,  a  guide  had  been  disseminated  to  participants  that  set  forth  the  ONA  process 
outlines  for  the  six  functional  PMESII  areas,  as  well  as  detailed  questions  about  building  a 
knowledge  base  across  those  categories.  Additionally,  the  multinational  participants 
received  the  latest  ONA  white  paper  and  tactics,  techniques,  and  procedure  documents, 
which  originated  from  Millennium  Challenge  2002. 

At  the  initial  planning  conference,  the  ONA  concept  brief  again  was  presented  to  a 
larger  body  of  MN  LOE  participants.  At  this  time,  a  plan  of  action  to  build  the  ONA  was 
also  presented  that  set  forth  a  building-block  approach  to  develop  the  ONA  knowledge 
base.  The  plan  included  a  series  of  ONA  workshops  and  homework  assignments  to  be 
completed  between  planning  conferences. 

ONA  Workshop  1  was  conducted  October  1-2,  2002,  on  site  at  the  General 
Dynamics  facility,  Suffolk,  Va.  The  workshop  began  with  an  azimuth  check  from  the 
multinational  participants  on  their  progress  in  developing  the  knowledge  base.  Then 
participants  received  training  on  the  system-of-systems  analysis  (SoSA),  an  analyst’s 
presentation  of  a  template  for  a  completed  PMESII  summary,  a  database  methodology, 
and  the  ONA  process.  To  build  upon  the  progress  of  Workshop  1,  “homework”  was 
assigned  to  participants  to  continue  the  knowledge  building  that  had  already  begun,  and 
specifically  to  complete  their  national  PMESII  summaries  and  to  identify  key  nodes  and 
critical  vulnerabilities  within  those  six  PMESII  categories. 

Workshop  2  was  held  November  5-7,  2002,  at  the  Renaissance  Hotel,  Portsmouth, 
Va.,  and  at  the  Joint  Futures  Lab,  Suffolk,  Va.  The  workshop  continued  the  education  process 
begun  in  Workshop  1.  Two  days  of  classes  and  demonstrations  discussed  the  ONA  concept 
and  provided  illustrations  of  the  SoSA  and  ONA  process,  a  database  familiarization,  and  a 
briefing  on  the  combined  multinational  PMESII  summaries.  The  afternoon  of  the  second  day 
included  a  seminar  on  “Blue-Red  Views,”  as  well  as  candidate  node  presentations.  On  the 
morning  of  the  third  day,  a  representative  of  the  ClearForest  commercial  software  company 
presented  a  demonstration  of  a  data-mining  tool  called  ClearResearch. 

In  the  intervening  period  between  ONA  Workshops  2  and  3,  ad  hoc  workshops  were 
conducted  to  support  newly  designated  multinational  participants.  A  cell  from  SACLANT 
was  introduced  late  in  the  planning  to  play  the  role  of  a  regional  nation  during  the  LOE  2. 
Due  to  personnel  tempo,  several  UK  analysts  were  identified  as  participants  late  in  the 
planning  process.  Two  separate  ONA  workshops  were  added  December  10-12,  2002,  and 
January  7-8,  2003,  to  provide  a  foundation  for  the  ONA  process,  the  system-of-systems 
approach,  PMESII  summaries,  database  methodology,  and  tools  training  and  demonstrations. 
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ONA  Workshop  3  occurred  January  15-16,  2003,  using  the  Groove  collaboration 
tool.  For  this  event,  multinational  participants  remained  in  their  home  nations,  and  the 
workshop  was  conducted  in  a  collaborative  information  environment.  This  workshop 
focused  on  ONA  process  refinement,  as  well  as  on  training  for  the  ONA  graphical  layout 
tool,  the  ONA  database,  and  the  ONA  geographic  visualization  tool.  This  was  the  last 
chance  for  ONA  training  until  MN  LOE  II  execution  in  Week  0. 

The  first  five  days  of  the  LOE,  Week  0,  offered  a  review  of  the  ONA  and  effects- 
based  operations  (EBO)  concepts  to  the  experiment  audience  as  a  whole.  In  addition, 
refresher  training  was  provided  for  the  Groove  collaboration  tool  and  the  ONA  database 
tools.  SoSA  business  process  rules  were  also  explained  in  order  to  provide  a  common 
understanding  of  the  mechanics  of  the  ONA  process.  Finally,  an  illustrative  linking  of 
effects,  nodes,  actions,  and  resources  (ENAR)  was  conducted,  as  well  as  a  combined  effort 
by  all  participants  to  review  the  database  inputs  to  date  and  to  reconcile  the  updates  and 
changes.  At  the  conclusion  of  Week  0,  the  ONA  knowledge  base  was  partially  complete;  a 
pool  of  effects,  nodes,  actions,  resources,  and  some  ENAR  linkages  was  available.  Figure  4 
demonstrates  the  major  emphasis  within  the  ONA  process  during  MN  LOE  II  execution. 


Figure  4.  MN  LOE  II  Execution  ONA  Process 


E.  INFORMATION  SHARING 

Analysis  based  on  participant  questionnaire  responses  and  analyst  observations 
indicates  no  significant  difference  between  the  two  infonnation-sharing  treatments. 

The  ONA  process  was  the  main  interest  of  the  partner  nations;  they  tended  to  view 
information  sharing  as  more  of  a  “U.S.  problem.”  The  ONA  process  was  new  and  more 
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interesting  to  them,  while  infonnation-sharing  problems  have  been  around  forever.  Thus, 
the  U.S.  placed  more  emphasis  on  presenting  the  ONA  process  in  an  “assembly”  forum.  The 
U.S.  analysts  and  planners  took  the  lead  as  participants  and  conducted  the  ONA  process  in 
much  the  same  way  as  in  Millennium  Challenge  2002,  a  U.S.-only  experiment  where 
discussions  occurred  in  a  virtual  assembly  area.  Releasing  national  infonnation  to  the 
coalition  was  common  in  Week  1,  and  observers  noted  that  some  participants  truncated 
national  disclosure  procedures.  The  net  result  was  that  the  current  infonnation-sharing 
treatment  was  practically  the  same  as  the  future  infonnation-sharing  treatment.  However, 
participants  noted  that  the  infonnation-sharing  mechanics  of  future  infonnation  sharing — 
where  all  infonnation  released  to  a  nation  was  available  in  the  coalition  and  national  shared 
spaces — potentially  enables  better  collaboration. 

In  addition,  the  injects  that  participants  received  at  the  beginning  of  each  vignette 
to  process  for  release  were  created  prior  to  the  CISP3  meeting,  so  their  decisions  did  not 
affect  many  of  the  disclosures.  Week  2  offered  no  injects  that  required  nations  to  exercise 
disclosure  procedures,  regardless  of  the  intelligence  exchange  requirements. 

Finally,  CISP3  unanimously  decided  to  list  exceptions  to  automatic  release  rather 
than  to  develop  explicit  items  for  automatic  release.  This  was  a  laudable  attempt  to  reduce 
the  administrative  burden  on  SoSAs  and  planners  as  they  detennined  if  a  piece  of 
information  could  be  released  without  going  through  a  formal  disclosure  process. 
However,  this  could  have  influenced  the  employment  of  the  future  information  sharing  by 
making  it  more  like  current  information  sharing. 

The  MN  LOE  II  was  the  first  opportunity  to  exercise  the  MNIS  OPCON.  The 
LOE  design  team  tried  to  represent  faithfully  the  intent  of  the  OPCON  within  the  LOE 
bounds,  while  not  significantly  affecting  the  ONA  portion  of  the  experiment.  The  LOE 
yielded  useful  insights  that  will  be  used  to  mature  the  MNIS  OPCON. 

F.  DATA  COLLECTION  AND  ASSESSMENT  METHODOLOGY 

The  assessment  team  was  organized  to  support  the  analysis  functions  of  the 
experiment:  assessment  planning,  data  collection,  data  analysis,  and  results  reporting.  All 
partner-nation  analysts  were  integrated  into  the  JFCOM  analysis  team  to  contribute  to  the 
assessment  process,  from  planning  to  reporting.  The  fellowship  and  excellent  professional 
and  working  relationships  among  the  members  of  the  experiment  analysts  working  group 
yielded  interactions,  discussions,  and  cultural  differences  that  fostered  and  contributed  to  the 
MN  LOE  II  program  and  to  its  resultant  success. 

The  hierarchy  of  data  flow  and  organization  for  the  experiment  analyst  team  that  was 
used  for  planning  and  execution  is  depicted  in  Figure  5.  Since  this  was  a  U.S. -led  experi¬ 
ment,  the  government  lead  and  the  analysis  division  chief  are  from  the  U.S.  Specific  billet 
descriptions  and  responsibilities  follow. 
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Process 
Observers. 

The  analysis 
process  relied 
upon  a  team 
of  observers 
who  were 
functional  ex¬ 
perts — mili¬ 
tary  and  civil¬ 
ian — in  one  or 
more  relevant 
areas.  They 
had  technical 
expertise  and 
experience  in 
a  particular 
service/joint 

military  operation,  unit,  organization,  weapon  system,  C4  system,  or  process  at  the 
tactical,  operational,  or  strategic  level  of  war.  Observers  were  culled  from  each  of  the 
coalition  partners,  based  upon  their  acknowledged  expertise  in  conducting 
observations  and  in  collecting  data. 

Observer  Lead: 

□  Directed,  coordinated,  and  interfaced  with  site  observers  regarding  data 
collection  and  responsibilities 

□  Discussed  issues  with  site  observers  to  provide  them  with  feedback  and  to 
solicit  their  comments  during  the  experiment 

□  Conducted  daily  feedback  meeting  with  site  observers 

□  Provided  hot-wash  observations  to  the  experiment  analyst  lead  via  Groove  e-mail 

□  Coordinated  the  timely  submission  of  surveys. 

Site  Observers: 

□  Followed  the  directions  of  the  lead  observer 

□  Submitted  insights  from  over-the-shoulder  data  collection  to  the  database;  ensured 
focused,  efficient,  complete,  and  accurate  collection  of  specific  data  elements 

□  Submitted  comments  and  recommendations  to  the  database  that  surpassed  the 
original  scope  of  survey  questions 

□  Ensured  the  timely  completion  of  surveys 

□  Ensured  that  the  product  remained  militarily  relevant  and  accurate. 

Experiment  Analysts.  Military  and  civilian  analysts  had  operations  research/systems 
analysis  training  or  practical  analytical  experience.  They  were  involved  in  all  phases 
of  the  analysis  process:  planning,  collecting,  analyzing,  and  reporting. 


Figure  5.  Experiment  Analysis  Organization 
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Analysis  Lead: 

□  Reported  to  the  experiment  director  and  the  analysis  division  chief 

□  Controlled  overall  assessment  planning,  execution,  and  reporting,  and  ensured 
that  collection  and  analysis  efforts  met  the  intent  of  the  experiment  director 

□  Developed  the  analysis  plan,  methodology,  collection,  analysis,  and  reporting 

□  Coordinated  with  partner  nations’  experiment  analysts 

□  Provided  information  to  the  after-action  review  lead. 

C4I  Coordinator: 

□  Reported  to  the  lead  experiment  analyst 

□  Coordinated  all  of  the  data  collection  efforts  for  the  ONA  database  and 
Groove,  ensuring  adherence  to  the  overall  data  collection  plan 

□  Helped  to  develop  the  analysis  plan  and  methodology 

□  Coordinated  the  analysis  of  data  regarding  the  C4I  systems. 

Survey  Coordinator: 

□  Reported  to  the  lead  experiment  analyst 

□  Administered  all  surveys  via  the  Joint  Battle  Center  (JBC)  Data  Collection  and 
Analysis  Tool7  (JDCAT),  ensuring  timeliness  and  minimizing  delinquency  rates 

□  Helped  to  develop  the  analysis  plan  and  methodology 

□  Collected  comments  and  recommendations  from  the  database 

□  Collated  survey  data 

□  Coordinated  timely  submission  of  surveys. 

Experiment  Analyst: 

□  Reported  to  the  lead  experiment  analyst 

□  Ensured  that  collection  and  analysis  efforts  met  the  intent  of  the  experiment  director 

□  Obtained  from  observers  the  practical,  military,  tactical/operational/strategic 
level  of  war  relevance  and  insights  for  the  collection  and  analysis  processes 

□  Helped  to  develop  the  analysis  plan  and  methodology 

□  Monitored,  reviewed,  and  ensured  accurate,  complete,  and  timely  data  collection. 


7  JDCAT  is  a  survey  data  collection  tool  developed  within  the  Joint  Battle  Center  (JBC)  at 
USJFCOM.  Version  3.1  was  used  for  MN  LOE  II. 
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1.  Data  Collection  Strategy 

Data  from  various  sources  was  needed  to  evaluate  adequately  the  objectives  and 
COIs  of  the  MN  LOE  II.  The  extensive  data  collected  included  each  partner  nation’s 
contributions  to  various  sections  of  the  collection  plan. 

a)  Collection  Plan 

The  experiment  generated  both  qualitative  and  quantitative  data  needed  to  gain 
perspective  of  the  ONA  process  and  MNIS,  specified  in  the  data  collection  matrix  in  Appendix  F. 

Qualitative  data  included  participant  and  observer  surveys,  over-the-shoulder 
observations  collected  by  designated  observers,  senior  concept  developers’  insights  and 
observations,  hot  washes,  and  after-action  reviews.  Surveys  used  scale  response  questions, 
and  provided  space  for  additional  comments,  which  were  grouped  by  topic  for  consideration  of 
their  impact  on  ONA  and  MNIS.  All  participants  and  observers  also  were  encouraged  to  submit 
comments  and  recommendations.  Additionally,  senior  concept  developers  from  each  coalition 
partner  nation  offered  insights  and  observations  of  human  factors  and  processes  during  several 
one-hour  sessions  each  week.  See  Appendix  D  for  more  infonnation  on  SCD  usage. 

Quantitative  data  included  ONA-database-access  review,  over-the-shoulder 
observations  collected  by  designated  observers,  and  outputs  from  Groove  about  collaboration 
proceedings.  An  audit  log  of  the  server  that  houses  the  ONA  database  was  used  to  review 
records  of  access  to  the  ONA  database,  revealing  users  and  changes  made.  Participant  access 
of  other  ONA  knowledge-base  components  was  also  reviewed.  Observers  used  over-the- 
shoulder  data  collection  forms  to  record  specified  events  of  interest. 

Data  collected  from  human  factors  observers  spanned  the  qualitative  and 
quantitative  spectrum  in  a  review  of  the  ONA  process.  Situational  awareness  probe 
questions  were  administered  to  all  players  at  all  multinational  sites  to  establish  the  task 
information  source  within  the  extended  multinational  ONA  team.  In  addition,  players’ 
workloads  were  detennined  during  the  multinational  ONA  development  process.  Next, 
organizational  interoperability  questions  covered  higher-level,  nontechnical  aspects,  such 
as  cultural  factors,  that  may  impede  such  a  working  cooperation. 

b)  Data  Management 

All  collected  data  was  accessible  to  all  participating  nations’  experiment  analysts, 
and  analysis  and  findings  were  shared  in  a  post-experiment  workshop.  As  part  of  the  post¬ 
experiment  management  plan: 

□  Senior  concept  developer  insights,  observations,  and  interview  results  were 
collected  and  maintained  in  Word  documents  and  in  JDCAT. 

□  Data  from  participants,  senior  concept  developers,  and  observer  surveys,  as 
well  as  comments  and  recommendations,  was  stored  in  JDCAT. 

□  Audit  log  data  from  the  ONA  database  server  was  transferred  to  an  Excel  spreadsheet. 
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□  User  products,  such  as  chat  sessions,  generated  via  Groove  during  execution, 
were  stored  and  maintained  in  a  Word  document  or  an  Excel  spreadsheet. 

□  The  audit  log  from  Groove  was  maintained  in  an  Excel  spreadsheet. 

□  The  situational  awareness  probe  and  the  workload  data  were  maintained  in  an 
Excel  spreadsheet. 


Figure  6.  Daily  Analysis  Battle  Rhythm 


c)  Analysis  Battle  Rhythm. 

The  analysis  battle  rhythm  was  built  upon  the  eight-hour  “Experimentation  Day.” 
See  Figures  6  and  7.  All  analysis  meetings  were  conducted  collaboratively  via  Groove. 

Assessment  Rally.  The  assessment  rally  was  a  daily  kickoff  meeting  to  review 
highlights  of  the  previous  day  and  to  give  assignments  for  the  day.  All  experiment 
analysis  personnel  attended  this  virtual  meeting. 

Flagged  Meeting.  At  any  time  during  the  experimentation  day,  analysts  could  report 
any  unusual  occurrences  in  an  impromptu  meeting  or  individual  conversation,  as  needed. 

Post-Vignette  Insights  and  Adjustments.  Also  during  assessment  rallies,  analysts 
gave  insights  and  recommendations  for  necessary  adjustments  following  each  vignette. 
They  were  prepared  to  answer  the  questions:  “What  did  we  learn  from  the  last  vignette?” 
and  “What  do  we  need  to  do  to  adjust  for  the  next  one?” 

Post-Hot  Wash  Meeting.  Analysts  participated  directly  in  hot  washes,  but  post-hot 
wash  meetings  were  not  required. 
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Post-Vignette  Insights  &  Adjustments 


Vignette  Vignette 


Vignette  Vignette 

3  4 


Figure  7.  Post-Vignette  Insights  and  Adjustments 


2.  Human  Factors  Plan 

A  human  factors  methodology  measured  changes  in  participants’  situational 
awareness.  Situational  awareness  is  a  person’s  perception  of  the  available  information 
and  his  understanding  of  the  immediate  situation,  his  projection  of  future  situation,  and  his 
understanding  of  how  to  resolve  the  problem.  A  measurement  of  the  participants’ 
awareness  of  the  shared  information  was  used  to  compare  the  effectiveness  of 
information- sharing  methods  in  the  experiment.  To  chart  the  development  of  situational 
awareness  over  time,  participants  responded  every  two  hours  to  five  probe  statements  as 
true  or  false.  Additionally,  participants  rated  their  need  to  know  a  certain  piece  of 
information,  their  actual  knowledge  of  it,  and  who  else  should  know  it.  The  probes 
presented  facts  regarding  the  current  scenario  but  did  not  reveal  infonnation,  breach  the 
information- sharing  boundaries,  or  bias  the  experiment. 

The  questionnaires  were  available  on  the  human  factors  Web  site  to  be  completed 
at  the  prearranged  time  (two,  four,  and  six  hours  into  the  experiment  day).  When  the 
human  factors  observer  at  each  site  announced  a  break  for  probe  questions  at  the  requisite 
time,  players  visited  the  Web  site  using  Internet  Explorer,  entered  their  call  sign  and  the 
question  set  number,  and  completed  the  questions. 

Each  probe  statement  required  four  responses: 

1)  Is  this  relevant  to  your  task/role?  (yes,  maybe,  no) 

2)  Is  this  true  or  false? 
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3)  What  is  your  level  of  confidence  in  the  true/false  response?  (very  low,  low, 
medium,  high,  very  high) 

4)  Which  other  countries  will  answer  this  probe  most  correctly?  (AUS,  CA,  GE, 
RE,  UK,  US) 

Participants  completed  the  questions  in  silence,  without  consulting  other  participants 
or  the  ONA  database.  They  also  did  not  discuss  the  questions  after  presentation.  If  a  player 
did  not  know  the  answer  to  a  question,  he  guessed  at  the  true/false  response  and  then 
indicated  he  was  guessing  by  marking  “very  low”  on  the  confidence  scale. 

The  probe  statements  were  compiled  from  baseline  infonnation  in  the  ONA 
database  and  in  the  vignette  brief,  and  from  information  that  had  been  added  as  new 
injects  during  the  experiment.  Two  “distracter”  probes  should  have  been  answered 
correctly  by  every  participant,  regardless  of  the  infonnation- sharing  condition  or  the 
release  of  information  from  other  nations.  The  remaining  three  probe  statements  were 
releasable,  subject  to  the  relevant  information-sharing  agreements. 

Each  probe  question  underwent  a  variety  of  checks  prior  to  its  use: 

1)  The  probe  questions  were  fonnulated  initially  from  the  inject  materials  and 
background  infonnation  to  test  knowledge  of  a  specific  piece  of  infonnation, 
without  giving  away  any  information  to  a  participant  who  had  not  read  the  inject. 
Equal  numbers  of  true  and  false  probe  statements  were  fonnulated,  ensuring  that 
false  statements  did  not  rely  solely  on  use  of  a  negative  in  the  statement. 

2)  Possible  probe  statements  were  submitted  to  three  different  criteria  tests: 

a)  Ten  individuals  with  no  knowledge  of  the  experiment  or  the  scenario  rated 
each  statement  as  either  true  or  false.  Since  the  probe  statements  were 
related  to  a  fictional  situation  in  2010,  this  group  answered  the  questions 
“blindly”  and  therefore  relied  on  the  structure  of  the  sentences  and  use  of 
language  to  guess  the  answer. 

b)  The  German  human  factors  lead  also  assessed  each  statement  for  its  level 
of  language  difficulty,  ensuring  that  the  nonnative-English  speakers  could 
understand  the  statements.  Each  statement  was  rated  for  “language 
difficulty”  on  a  scale  of  1-5. 

c)  An  intelligence  specialist  observer  from  the  UK  white  cell  reviewed  the 
probe  statements  for  operational  significance  on  a  scale  of  1-3. 

3)  The  final  selection  of  possible  probe  questions  included  only  those  statements 
that  received  an  equal  rating  of  true  and  false  responses  in  a)  above,  a  score  of 
1-3  in  b)  above,  and  a  rating  of  1-2  in  c)  above. 

4)  These  statements  were  submitted  to  the  analyst  lead  at  JFCOM  in  the  United 
States  for  final  approval  and  then  were  considered  suitable  for  use  with  the 
participants. 

The  timetable  of  probe  statements  contained  equal  numbers  of  statements  asked 
about  information  released  to  each  nation.  Each  statement  from  experiment  day  one  was 


27 


repeated  between  presentations  in  order  to  assess  the  amount  of  growth  in  understanding 
during  a  set  time  period.  Furthermore,  each  presentation  assessed  infonnation  in  different 
ONA  areas — nodes,  actions,  resources,  or  effects — and  different  nations  of  interest  in 
order  to  balance  the  individuals  who  should  answer  each  statement  correctly. 

3.  Assumptions 

During  the  planning  and  execution  of  MN  LOE  II,  some  assumptions  were  made 
that  could  have  affected  the  conduct  of  the  experiment  and  resulting  analysis.  As  in  any 
experiment  or  analyses,  these  fundamental  assumptions  were  required  for  the  successful 
execution  of  the  experiment: 

□  Personnel  have  the  requisite  education,  training,  and  experience  to  perform 
their  duties. 

□  Sufficient  training  would  be  available  to  assigned  personnel  to  allow  them  to 
become  familiar  with  the  concepts  and  tools  to  be  used  during  the  experiment. 

□  The  political  climate  of  the  region  depicted  in  the  2010  scenario  was  selected. 

□  Week  1  of  the  experiment  depicted  the  current  method  of  infonnation  sharing. 

□  Week  2  of  the  experiment  depicted  the  future  method  of  information  sharing. 

□  The  use  of  experiment  injects  would  result  in  a  sufficient  amount  of 
information  sharing  to  detennine  differences  between  Week  1  and  Week  2. 

4.  Limitations 

During  the  planning,  execution,  and  analysis  phases  of  the  experiment,  all  of  the 
qualities  and  requirements  of  MN  LOE  II  were  combined  to  meet  the  experiment  objectives, 
resulting  in  recognized  limitations  that  could  influence  the  conduct  of  the  experiment  and  the 
validity  of  the  results.  Some  limitations  grew  out  of  existing  technical  shortfalls,  while  others, 
sometimes  referred  to  as  delimiters ,  intentionally  produced  specific  opportunities  to  explore 
the  concepts  and  to  meet  the  experiment  objectives.  Still  others  were  used  to  balance  and 
prioritize  diverse  requirements.  Consequently,  the  existence  of  limitations  does  not  mean 
necessarily  that  they  had  a  negative  impact  on  the  validity  of  the  experimental  findings. 

These  limitations  exemplify  the  difficulty  of  replicating  any  scenario  in  an 
experimentation  environment.  Ideally,  the  experiment  concepts  would  have  been  applied 
within  the  experiment  exactly  as  the  concept  developers  envisioned  them.  However,  this 
was  not  always  so.  Concepts  are  futuristic  by  their  very  nature,  and  in  some  instances, 
they  are  ahead  of  available  technology.  Thus,  the  assessment  of  a  concept  is  relevant  to 
the  degree  to  which  it  is  accurately  applied  in  the  experiment. 
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These  limitations  emerged  during  the  MN  LOE  II  planning,  execution,  and  analysis: 

□  The  experiment  was  executed  using  six  hours  of  operation  per  day  instead  of 
24  hours  of  operation  per  day. 

□  Assessors  used  some  intrusive  means  to  collect  data. 

□  A  lack  of  a  command  structure  existed  within  the  participating  nations.  In 
some  cases,  this  resulted  in  a  lack  of  national  guidance  and  direction  that 
detracted  from  the  experiment  objectives. 

□  Limited  access  to  infonnation  sources  outside  of  the  experiment  during  the 
execution  period  detracted  from  the  ONA  process. 

□  A  lack  of  detailed  procedures  existed  for  the  execution  of  the  ONA  process  in  a 
multinational  environment.  Those  that  eventually  were  developed  and 
promulgated  were  deemed  acceptable,  but  only  for  the  experiment.  Further 
work  is  required  in  this  area. 

□  MN  LOE  II  was  a  single,  one-time  experiment  without  multiple  trials. 

□  Differences  existed  between  the  actual  concepts  and  the  way  they  were 
operationalized  in  the  experiment. 

□  Participant  training  and  knowledge  of  the  concepts  was  less  than  may  be 
expected  upon  fielding  of  the  concepts. 

□  The  ONA  tool  was  marginally  populated  with  information  at  the  beginning  of 
the  experiment. 

□  Specific  statistical  results  of  the  study  will  hold  true  only  for  scenarios  with  a 
similar  construct.  Inferences  drawn  from  the  statistics  are  related  to  the 
concepts  tested. 

□  The  scenario  was  notional  and  occurred  in  the  future. 

□  The  security  domains  tested  within  the  experiment  were  notional  within  the 
Groove  network.  Infonnation  was  not  shared  across  real-world  security  domains. 

5.  Assessment  Methodology 

All  data  collected  was  broken  down  into  quantitative  or  qualitative  categories. 
Qualitative  data  was  analyzed  for  trends  and  commonalities  in  opinions  and  insights  and 
for  differences  in  rating  metrics  produced  by  the  given  scales.  Quantitative  data  was 
analyzed  for  time  and  event  frequencies  of  the  ONA  planning  process.  The  sources  for  all 
qualitative  and  quantitative  data  are  detailed  for  each  objective  in  Table  2.  For  this 
experiment,  qualitative  data  is  subjective  appraisals  of  events  by  participants  and 
observers,  distilled  with  their  experience  and  judgment.  Quantitative  data  is  objective 
measurements  of  events  from  nonjudgmental  observers  or  from  instrumentation. 

The  quantitative  source  of  analyzed  data  is  the  ONA  knowledge  base  activity  recorder. 
An  audit  log  of  the  Structured  Query  Language  (SQL)  server  that  housed  the  ONA  database 
was  used  to  collect  participant  access  to  the  stand-alone  ONA  relational  database.  This  log 
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identified  the  user  and  changes  made.  A  Groove  log  also  was  used  to  collect  infonnation  on 
Groove  tool  and  space  usage.  To  clarify  findings  as  they  emerged,  all  of  these  collected  data 
were  matched  against  the  COIs  that  supported  the  experiment’s  two  main  objectives. 


Objective  1 
ONA  in  MN 
environment 

Qualitative  Data 
(Subjective) 

Web-Based  Surveys 
-  Rating  scale 

(SCD,  observer,  participant) 

+ 

Opinions  &  Insights 

-  Survey  additional  comments 

-  Interviews 

-  Comments  and  recommendations 

-  AAR/hot-wash  minutes 
(SCD,  observer,  participant) 

Quantitative  Data 
(Physical  Measures) 

Frequency  of  database  accesses 
and  changes 
(C4I  systems) 

Objective  2 
Info  Sharing 

Week  1  vs.  Week  2: 

Week  1  vs.  Week  2: 

Across 

Web-Based  Surveys 

ONA  SOL  server  audit  log 

Security 

-  Rating  scale 

(access/changes) 

Domains 

(SCD,  observer,  participant) 

+ 

+ 

Over-the-shoulder  observations 

Opinions  &  Insights 

(Observer  data  sheets) 

-  Survey  additional  comments 

+ 

-  Interviews 

Groove  server  audit  log 

-  Comments  and  recommendations 

(Tool  and  space  usage) 

-  AAR/hot-wash  minutes 

+ 

(SCD,  observer,  participant) 

Targeted  questions  about  SA 
Perceptions  vs.  ground  truth 
(participant) 

Table  2.  Qualitative  and  Quantitative  Data  for  Objectives  1  and  2 


The  assessment  methodology  differed  slightly  for  the  two  objectives: 

Methodology  for  Objective  1.  Qualitative  data  was  used  to  assess  the  workability 
of  the  ONA  process  in  a  multinational  environment.  The  analysis  was  not  based  on  a 
comparison,  but  on  whether  the  process  works. 

Methodology  for  Objective  2.  Both  qualitative  and  quantitative  data  was  used  to 
compare  current  with  future  methods  of  information  sharing.  Targeted  questions 
concerning  situational  awareness  also  were  analyzed  for  differences  between  perceived 
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and  ground  truth.  The  analysis  here  was  based  on  comparisons  of  performance  measures 
for  each  information-sharing  method.  Qualitative  and  quantitative  results  should  agree. 

Furthermore,  an  experiment  analysis  workshop  was  convened  to  enable  all  analysts 
to  contribute  their  inputs  to  the  analysis  process.  Partners  came  prepared  to  discuss  their 
insights  and  to  present  their  independent  analysis  of  the  experiment.  For  the  results  of  this 
workshop,  see  Appendix  A. 
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IV.  MAIN  EXPERIMENT  FINDINGS 


A.  OBJECTIVE  1  DISCUSSION 

Identify  and  assess  issues  associated  with  the  ability  of  national  headquarters  staffs 
to  conduct  a  distributed  ONA. 

1.  Objective  1  Overall  Assessment  Results 

The  MN  LOE  II  demonstrated  that  a  multinational  coalition  could  successfully 
conduct  the  U.S.  ONA  process  in  a  distributed  environment  under  controlled  experimental 
conditions.  The  mechanics  and  procedures  of  the  process  worked  well,  except  for  a  few 
needed  refinements.  Some  more  advanced  issues  need  resolution  that  requires  a  better 
examination  of  national,  non-U. S.  perspectives. 

Major  problems  included  time  constraints  that  prevented  significant  exploration  of 
national  views,  a  lack  of  national  command-and-control  organizational  structure  to  allow 
consideration  of  true  national  postures,  and  a  lack  of  conflict  resolution  ability  in  a 
collaborative  environment.  These  kinds  of  concerns — at  the  heart  of  multinational 
cooperation — began  to  surface  in  collaborative  discussions  but  were  not  explored.  Until 
they  are  explored  more  fully,  the  ability  of  nations  to  collaborate  cannot  be  fully  appraised. 
Thus,  even  though  experiment  findings  suggest  the  U.S.  ONA  process  is  compatible  with 
other  European-based  cultures,  higher-order  cultural  and  organizational  issues — also  central 
to  multinational  coalition  operations — need  further  exploration  in  future  experiments. 

Benefits  from  multinational  partners  included  a  wider  range  of  options  proposed  to 
achieve  objectives,  few  language  barriers,  and  overall  willingness  to  participate  in  a 
coalition  operation.  Lingering  issues  persist  in  two  major  areas.  First,  if  nations  disagree 
on  the  selection  of  PMESII  options,  how  could  this  conflict  be  managed  and  resolved  to 
preserve  the  coalition  objectives?  Second,  how  will  nations  define  the  architecture  of  their 
own  “ONA  organization”  and  its  links  to  their  national  agencies,  and  how  will  they 
establish  the  crosswalk  between  elements  of  their  ONA  organization  and  corresponding 
elements  of  other-nation  ONA  organizations?  Without  the  opportunity  to  explore  these 
questions,  conclusions  drawn  about  cultural  and  organizational  factors  that  affect  a 
multinational  ONA  process  are  rudimentary  at  best. 

2.  COI  1.1  Discussion  -  Is  the  U.S.  ONA  process  viable  in  a  coalition 
environment? 


a)  COI  1.1  Assessment  Results 

With  the  appropriate  tools  and  infonnation-sharing  policies  and  procedures,  the 
ONA  process  can  be  performed  in  a  coalition  environment.  Overall,  the  ONA  process 
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was  well  received  by  experiment  participants,  who  considered  it  to  be  viable  in  a  coalition 
environment.  Adequate  training  on  the  process,  tools,  and  standard  operating  procedures 
is  essential;  staff  availability  and  use  of  the  distributed  network  presented  training 
obstacles.  Even  within  the  same  nation,  training  levels  of  players  varied. 

Most  player  survey  responses  indicated  positive  reactions  to  the  ONA  process, 
with  reservations  expressed  within  these  general  comments: 

1)  A  more  robust  voice  communication  capability  is  needed  to  support 
collaboration. 

2)  Collaboration  procedures  must  be  structured  more  effectively. 

3)  The  ONA  process  was  not  fully  tested  for  multinational  use. 

4)  The  role  of  national  interests  must  be  addressed  to  a  greater  degree. 

5)  Consensus  and  conflict  resolution  were  not  explored  sufficiently. 

6)  Other  nations  perceived  a  U.S.  bias  toward  military  approaches  to  problems. 

These  general  reservations  comprise  those  that  pertain  to  lower-level,  easily  resolved 
issues,  such  as  mechanical  and  procedural  concerns,  and  those  that  pertain  to  higher-level 
issues,  such  as  national  perspective,  that  are  at  the  heart  of  the  multinational  arena.  Given 
the  limited  experiment  goals  of  demonstrating  and  exploring  the  basic  ONA  process  in  a 
multinational  setting,  the  process  was  shown  to  be  viable  under  the  experiment  conditions. 

b)  COI  1.1  Findings 

(1)  Finding  1  ->  All  data  considered,  the  experiment  participants 

indicated  that  the  basic  ONA  process  is  viable  in  a  coalition  environment. 

The  majority  of  survey 
respondents  across  nations  agreed  that 
the  U.S.  ONA  process  is  viable  in  a 
coalition  environment.  (See  Figures  8 
and  9.)  Due  to  the  departure  of  some 
players,  the  total  number  of  responses 
decreased  slightly  from  Week  1  to 
Week  2.  In  addition,  the  percentage  of 
participants  who  agreed  dropped  from 
90  percent  in  Week  1  to  73  percent  in 
Week  2.  This  decrease  may  be  due  to 
participants’  better  understanding  of  the 
complexity  of  the  ONA  process  by  the 

end  of  the  experiment.  Nevertheless,  ...  0  .  ....  .... 

1  ’  Figure  S.  The  U.S.  ONA  process  is  viable  in  a  coalition 

the  percentage  of  positive  responses  for  environment  (Week 
the  two  weeks  is  still  markedly  higher 
than  the  percentage  of  negative  responses. 
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During  the  experiment, 
players  who  perfonned  the  ONA 
process  quickly  grasped  the  basic 
mechanics  and  expressed  a  drive  to 
move  forward  to  tackle  more 
sophisticated  issues.  These  results 
emerged  despite  a  plague  of  voice 
communication  problems  with  the 
collaborative  tool  used  in  the 
experiment.  The  basic  process  was 
feasible  across  continents  under  these 
poor  circumstances,  underscoring  its 
potential  in  more  optimal  settings. 

(2)  Finding  2 

The  ONA  process  re¬ 
quires  a  robust,  distrib¬ 
uted,  collaborative  envi¬ 
ronment,  including  voice,  text,  and  visuabzation  capabilities. 

Various  tools  are  needed  to  effectively  facilitate  distributed  multinational  collaboration. 
Participants  complained  that  the  collaborative  tool  used  in  the  experiment  had  an  inadequate 
voice  capability  that  could  not  support  collaboration  by  a  large  number  of  players. 

Most  participants  blamed  the  Groove  tool  specifically,  while  others  pointed  to 
inadequate  bandwidth  allocated  on  the  CFBL  net;  the  source  of  the  communication 
problems  is  unclear.  But  clearly,  ONA  must  be  supported  with  a  secure,  reliable,  robust, 
flexible  collaborative  tool  or  tools  capable  of  supporting  large  collaboration  groups.  The 
problem  was  exacerbated  by  the  artificial  constraints  on  the  use  of  the  Groove 
tool/capability  due  to  the  collaboration  business  rules. 

The  ONA  database  tool  was  deemed  cumbersome  and  not  particularly  friendly,  which 
led  to  the  observation  that  the  experiment  became  more  of  an  exercise  in  feeding  and 
maintaining  the  database  than  in  exploring  the  ONA  process  and  its  potential  to  assist 
decision-makers  in  gaining  insights  and  knowledge. 

The  problem  was  mitigated  somewhat  by  procedurally  limiting  the  number  of 
speakers  and  by  using  interactive  text.  But  this  slowed  communication  and  caused  players 
frustration  and  confusion  by  losing  track  of  conversations  conducted  in  text  mode.  This 
particular  concern  could  have  been  resolved  through  a  threaded  discussion  capability. 

Some  players’  comments  included  more  than  simple  tool  inadequacies,  raising  the 
general  subject  of  future  equipment  interoperability  across  26  NATO  nations. 


Figure  9.  The  U.S.  ONA  process  is  viable  in  a  coalition 
environment  (Week  2). 
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(3)  Finding  3  ->  Collaboration  business  rules  must  be  defined  in 
detail,  taught  in  advance,  and  practiced  regularly  to  support  successful 
multinational  collaboration. 

Business  rules  were  poorly  defined  prior  to  the  start  of  the  experiment  and  lacked 
sufficient  detail  to  adequately  support  the  coalition  collaboration  process.  They  continued 
to  evolve  and  were  refined  during  the  three  trial  weeks. 

As  the  experiment  began,  the  participants  followed  a  U.S. -developed  ONA  concept 
and  process,  which  the  business  rules  supported.  As  the  ONA  process  became  more 
multinational,  the  business  rules  accommodated  those  process  refinements.  Further,  the 
rules  were  adapted  for  the  technical  limitations  of  the  collaboration  tool  suite  and  of  the 
ONA  database.  Participants  noted  that  implementation  of  a  notional  command  structure  in 
each  of  the  participating  nations  would  have  supported  the  experimental  play  and  could 
have  solved  some  of  the  problems  in  business  rule  definition  and  implementation. 

In  addition,  a  trained,  effective,  and  experienced  moderator  or  session  leader  is 
critical  to  the  smooth  and  efficient  conduct  of  collaborative  sessions. 

(4)  Finding  4  Further  testing  of  the  ONA  process  is  needed 
within  the  multinational  arena. 

Comments  suggested  that  the  experiment  did  not  explore  some  elements  of  ONA 
viability  in  a  multinational  enviromnent.  Participants  said  that  the  experiment  explored  only  the 
mechanical  and  basic  procedural  requirements  to  conduct  the  process.  Comments  included: 

□  Multinational  communications  and  a  fairly  primitive  decision-making  tool 
were  tested  and  found  to  be  just  above  adequate. 

□  How  scalable  is  the  ONA  process — i.e.,  how  many  nations/organizations  can 
participate  in  the  process  before  it  becomes  unworkable  or  before  the  results 
deteriorate? 

□  Participants  spent  all  of  their  time  developing  the  procedures  to  try  to  get  to  the 
concept;  they  did  not  think  that  they  ever  got  there. 

□  Instead  of  experimenting  with  multinational  ONA  process  procedures, 
participants  developed  them  during  the  experiment.  This  limited  the  ability  to 
truly  assess  viability. 

□  Three  nations  (CA,  DE,  and  UK)  were  separately  exploring  their  own  effects- 
based  operations  processes  and  thus  had  a  different  opinion  of  how  ONA 
should  work. 
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(5)  Finding  5  ->  Incorporation  of  national  guidance  and  objectives  is  needed. 

The  issue  of  national  perspectives  was  not  played  during  the  experiment,  although 
participants  and  SCDs  frequently  cited  it  as  being  central  to  determining  ONA  viability. 
Players  were  unsatisfied  with  the  ability  to  coordinate  a  national  position  within  the  ONA 
process.  Under  the  current  information- sharing  method,  a  synthesis  of  different  national 
procedures  and  a  multinational  linking  of  nationally  shaped  information  are  needed. 
Participants’  comments  included: 

□  The  process  needs  considerable  work,  including  a  realistic  way  of  capturing 
national  positions/perspectives,  which  is  critical. 

□  Incorporation  of  national  perspectives  at  a  greater  level  would  improve  the 
quality  of  the  data  and  linkages  incorporated  in  the  ONA. 

□  We  need  a  multiagency  approach,  including  government  representatives 
empowered  to  make  decisions  on  government  policy. 

□  Some  type  of  commander  role  is  needed  within  each  nation  to  structure  the 
interactions  from  a  national  perspective  to  maintain  sovereign  positions. 

(6)  Finding  6  ->  Consensus  and  conflict  resolution  were  not 
explored  sufficiently. 

According  to  participants  and  SCDs,  conflict  resolution  and  consensus-building 
among  nations  during  collaboration  also  require  more  in-depth  examination.  These  issues 
received  some  play,  but  unfortunately  were  abbreviated  due  to  time  constraints.  Along 
with  the  issue  of  national  perspectives,  these  two  primary  issues  for  multinational  viability 
went  unapprised. 

Comments  included: 

□  “Real  issues  related  to  debates  and  conflict  resolution  [were]  not  addressed.” 

□  “The  issues  to  date  have  not  been  contentious  enough  to  really  test  the 
decision-making  system.” 

□  “There  must  be  an  emphasis  on  the  ‘coalition’  part — joint  decisions,  made  by 
consensus,  that  work  on  behalf  of  the  entire  coalition.” 

□  “[The]  U.S.  ONA  process  can't  run  over  the  national  decision-making 
processes.” 

□  “There  needs  to  be  specific  attention  paid  to  international  recommendations.” 

□  “Participating  countries  do  not  fundamentally  agree  on  the  nature  of  this  situation: 
[The]  U.S.  has  an  emphasis  on  the  military  aspects  ...  other  countries  on 
diplomatic  and  economic  aspects  ...  A  philosophy  of  ‘agree  to  disagree’  is  not 
appropriate  for  having  a  coalition  of  countries  working  together  to  make 
decisions.” 
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(7)  Finding  7  ->  Because  of  time  constraints,  the  experiment  did 

not  fully  explore  all  phases  of  the  spectrum  of  conflict  or  diplomatic, 
information,  military,  and  economic  (DIME)  actions,  and  chose  to  focus 
on  military  actions. 

This  perception  is  based  on  these  specific  trends  in  player  comments: 

□  U.S.  players  made  pronounced  attempts  to  deemphasize  military  solutions 
during  collaboration  with  other  nations,  but  the  bias  was  perceived  by  the 
partners  anyway. 

□  Excellent  discussions  concerning  shaping  effects  occurred  on  the  first  day  of 
Week  2  in  Vignette  3  among  multinational  planners,  SoSAs,  and  U.S.  planners. 
U.S.  planners  indicated  that  the  “worst  case”  military  effects  should  be 
discussed  first.  Multinational  partners  disagreed,  saying  that  the  emphasis 
should  be  diplomatic  and  economic. 

□  An  entity  like  the  joint  interagency  control  group  clearly  was  needed  to  address 
diplomatic  and  economic  aspects  among  partners.  International  partners 
preferred  ENAR  linkages  that  addressed  political  and  economic  actions, 
whereas  the  comfort  zone  of  the  U.S.  was  deliberate  military  planning. 

□  U.S.  planners  exceeded  the  intent  of  the  experiment  objectives  and  planned  for 
military  operations.  They  did  not  explore  the  diplomatic,  information,  and 
economic  options,  except  as  support  to  the  military  operations.  U.S.  planners 
need  to  broaden  their  thinking  beyond  their  military  comfort  zone. 

□  Even  the  U.S.  names  of  the  phases  of  the  spectrum  of  conflict — influence, 
deter,  coerce,  compel,  defeat,  instead  of  negotiate  or  entice — were  perceived 
by  the  partners  as  completely  militaristic. 

3.  COI  1.2  Discussion  -  What  are  the  impacts  of  cultural  and/or 
organizational  differences  on  coalition  collaboration? 

a)  COI  1.2  Assessment  Results 

Cultural  differences  resulted  in  a  positive  contribution  to  coalition  collaboration 
and  ONA  product  quality.  Different  cultural  perspectives  brought  to  the  ONA  process  a 
wider  range  of  potential  options  and  solutions.  However,  even  though  all  but  two  of  the 
participating  partner  teams  were  composed  entirely  of  native  English  speakers,  the  use  of 
slang  terms,  military  jargon,  acronyms,  and  different  usage  and  definitions  for  the  same 
words  in  the  different  nations  resulted  in  occasional  misunderstandings. 

In  a  coalition  with  many  more  members  or  one  in  which  participants  had  more 
diverse  political,  military,  and  social  customs,  the  degree  of  cooperation  in  this  LOE  could 
be  more  difficult  to  achieve.  The  presence  of  more  nations  increases  the  quantity  of 
information  available  and  thus  the  complexity  of  collaboration  and  coordination. 
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An  information- 
sharing  “culture”  must  be 
cultivated  in  each  of  the 
partner  nations.  Person¬ 
nel  must  be  well 
prepared  by  education, 
training,  and  experience 
to  deal  with  the 
complexity  or  diversity 
of  issues  that  typify  the 
ONA  process  and  the 
effects-based  approach  to 
operations  that  it 
supports. 

Coalition  partners 
remarked  on  the  U.S. 
dominance  of  the  ONA 
development  process, 
which  resulted  in  a  lack 
of  shared  meaning  of 
fundamental  terms.  Co¬ 
alition  partners  also  noted 
the  U.S.  proclivity  to 
jump  to  military  actions 
before  fully  exploring  all 
diplomatic,  economic,  and 
political  options.  U.S. 
comments  supported  this  Figure  11.  Impact  of  Cultural  Differences  on  Database  Quality 
dominant  position,  since 

ONA  was  a  U.S. -developed  tool.  Such  an  approach  potentially  may  affect  multinational 
collaboration  negatively. 

Each  partner  nation  implemented  its  ONA  cell  at  different  levels  in  its  national 
military  organization,  resulting  in  a  slightly  negative  impact  on  the  coalition  collaboration 
process  and  on  ONA  product  quality.  Difficulties  were  manifested  primarily  in  the 
different  manning  levels  for  the  partners,  which  did  not  always  allow  them  to  have 
expertise  in  all  of  the  required  analysis  and  planning  disciplines,  and  which  did  not  allow 
them  to  participate  fully  in  all  of  the  required  collaborations. 

In  order  for  ONA  to  be  implemented  successfully  in  a  multinational  collaborative 
environment,  each  partner  nation  must  determine  its  own  internal  ONA  structure  and  must 
develop  an  understanding  of  the  differences  in  organizational  structure  and  function. 
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Figure  10.  Impact  of  Cultural  Differences  on  Collaboration 
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b)  COI  1.2  Findings 


(1)  Finding  1  ->  Cultural  differences  had  an  overall  positive 
impact  on  coalition  collaboration  and  on  ONA  product  quality. 

The  coalition  member  nations  in  this  experiment  share  a  European-oriented  cultural 
background,  similar  military  traditions,  a  long  history  of  working  together,  and  quickly 
created,  trusting,  and  cooperative  working  relationships.  Participants  perceived  that  the 
differences  in  national  cultures  had  an  overall  positive  impact.  Players  believed  that, 
because  of  cultural  similarities,  commonwealth  countries  generally  worked  well  together 
and  supported  each  other.  Differences  between  civilian  and  military  cultures  potentially 
could  have  greater  impact  than  differences  among  partners’  military  cultures.  Some  partners 
preferred  a  more  rigid,  structured  approach  to  the  collaboration  process;  some  preferred  a 
more  flexible,  loosely  structured  approach.  See  Figures  10  and  1 1  for  the  survey  results. 

Participants  observed  that  cultural  input,  disagreements,  and  agreements  reduced  and 
discouraged  projecting  partner  values  onto  those  of  the  potential  adversary.  Encouraging 
multiple  views  from  different  cultural  perspectives  helped  to  avoid  the  tendency  to  believe 
that  an  adversary  thinks  like  we  do,  and  then  to  be  surprised  when  he  does  not. 

(2)  Finding  2  ->  A  common  language  and  use  of  common  terms 
are  required  for  successful  collaboration. 

Misunderstandings  that  resulted  from  use  of  slang,  jargon,  acronyms,  and  even 
common  terms  hindered  communication  somewhat.  Most  significantly,  national  partners 
understood  the  U.S.  ONA  tenns  effects,  nodes,  and  actions  differently.  As  one  regional  player 
observed,  “Language  differences  hindered  the  speed  at  which  you  [could]  work  and  the  degree 
of  understanding  you  could  achieve  on  certain  matters.”  The  participants  used  six  different 
versions  of  English,  and  their  range  of  civilian  and  military  backgrounds  led  to  gaps  in 
assumed  knowledge  and  differences  in  the  tenninology  used.  For  example,  a  U.S.  team 
member  commented  that  he  was  “not  familiar  with  all  military  tenns,”  and  another  player 
offered  that  “...there  are  distinct  jargons  for  military  personnel,  systems  engineers,  civilian 
area  specialists.  It  is,  on  occasion,  difficult  to  recognize  all  of  these  as  being  in  English.” 

Open  discussion  by  the  participants  and  implementation  of  effective  business  rules 
overcame  these  misunderstandings.  However,  a  permanent  awareness  of  the  presence  of 
non-English-speaking  participants  is  necessary.  Language  accommodation  skills  are 
required,  such  as  speaking  slowly,  not  using  unusual  terms  or  complex  phraseology,  and 
allowing  time  and  opportunity  to  clarify  questions  and  answers.  Incorporation  of  a  near- 
real-time  translation  capability  may  alleviate  some  of  these  concerns. 

(3)  Finding  3  ->  Collaboration  may  be  more  difficult  when  the 
group  is  larger  and  more  culturally  diverse  than  that  of  this  experiment. 

Participants  noted  that,  in  a  coalition  with  many  more  members,  or  one  in  which 
participants  had  a  wider  variety  of  political,  military,  and  social  customs,  the  degree  of 
cooperation  that  was  observed  in  this  LOE  would  be  more  difficult  to  achieve.  The 
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presence  of  more  nations  increases  the  quantity  of  infonnation  available  and  the 
complexity  of  collaboration  and  coordination. 

(4)  Finding  4  ->  A  philosophy  to  share  information  to  the  maximum 
extent  possible  must  be  cultivated  among  all  nations  within  the  ONA 
process. 

A  philosophy  of  collaboration  that  rewards  sharing  should  be  the  standard  rather 
than  the  exception  within  organizations  and  governments,  as  well  as  among  nations.  The 
tendency  to  restrict  the  disclosure  of  possibly  sensitive  information  should  be  avoided. 
Withholding  infonnation  must  be  minimized  in  a  multilateral  collaborative  environment. 

(5)  Finding  5  ->  U.S.  dominance  of  the  collaborative  process 
potentially  could  affect  coalition  collaboration  negatively. 

Coalition  partners  remarked  on  the  U.S.  dominance  of  the  ONA  development 
process  and  on  the  U.S.  proclivity  to  jump  to  military  actions  before  diplomatic, 
economic,  and  political  options  had  been  considered.  U.S.  comments  supported  this 
dominant  position,  since  ONA  was  a  U.S. -developed  tool.  Participants  indicated  that 
some  nations  eventually  “gave  up”  suggesting  nonmilitary  actions  because  the  U.S. 
repeatedly  argued  them  down,  leading  them  to  feel  that  their  input  was  “useless.” 

These  comments  reflect  previous  research  showing  that  U.S.  officers  are  less 
knowledgeable  about  other  cultures  than  are  their  multinational  counterparts  .  Most 
players  had  received  no  formal  training  on  cultural  differences  and  awareness.  Such  an 
approach  potentially  may  affect  multinational  collaboration  negatively. 

(6)  Finding  6  ->  Organizational  differences  had  a  slightly  negative 
impact  on  the  coalition  collaboration  process  and  on  ONA  product  quality. 

Each  partner  nation  implemented  its  ONA  cell  at  different  levels  in  its  military 
organization.  Nations  involved  in  the  experiment  used  different  command-and-control 
structures  and  processes  that  may  influence  some  partners’  ability  to  act  in  the 
collaboration  process. 

Organizational  difficulties  were  manifested  primarily  in  partners’  manning  levels, 
which  did  not  always  provide  expertise  in  all  required  analysis  and  planning  disciplines, 
and  which  did  not  allow  them  to  participate  fully  in  all  of  the  required  collaborations. 
Although  the  difference  in  manning  levels  may  have  been  an  artificiality  of  the 
experiment,  many  believed  that  real-world  implementation  still  would  result  in 
significantly  higher  staffing  levels  for  the  U.S.  and  potentially  could  result  in  a  U.S.- 
dominated  process,  particularly  during  precrisis  operations.  If  the  outcome  is  to  be  based 
on  a  consensus,  then  undermanned  national  teams  would  need  additional  time  to  review 
collaborative  work  in  which  they  were  not  able  to  participate. 

g 

Bowman,  Elizabeth  K.,  and  Pierce,  Linda  G.  2002.  Cultural  barriers  to  teamwork  in  a  multinational 
coalition  environment.  Poster  presentation  at  Army  Science  Conference,  December  2002. 
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Canada’s  participants 
commented,  “Among  the 
challenges  ahead,  partner 
nations  will  need  to  develop 
understanding  of  differences 
in  organizational  structure 
and  function,  which  result  in 
the  placement  of  partner’s 
ONA  capabilities  at  different 
levels  in  their  governments.” 
This  drives  the  requirements 
for  policies  and  procedures  to 
support  coalition  collabora¬ 
tion  and  information  sharing. 

National  perspectives 
generate  conflicting  choices 
for  DIME  options  in  seeking 
to  satisfy  a  common  coalition 
objective.  This  will  compli¬ 
cate  the  ONA  process,  but 
ideally,  it  will  lead  to  superior 
coalition  decision  support  and 
planning.  Figures  12  and  13 
depict  the  perceived  impact  of 
organizational  differences. 
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Figure  12.  Impact  of  Organizational  Differences 
on  Collaboration 
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Figure  13.  Impact  of  Organizational  Differences 
on  Database  Quality 


(7)  Finding 

7  ->  Each  participating  nation  must  determine  its  own  internal  ONA 
structure. 


In  order  for  ONA  to  be  implemented  successfully  in  a  multinational  collaborative 
environment,  each  nation  must  determine  its  own  internal  ONA  structure  and/or  similar 
process,  since  each  partner’s  ONA  capabilities  are  likely  to  reside  at  different  levels  in  its 
government.  Each  partner  nation  must  understand  of  these  differences  in  organizational 
structure  and  function. 


Access  to  other  government  agencies  is  central  to  the  implementation  of  ONA  for 
each  nation,  as  well  as  the  establishment  of  links  to  other  resources,  academia,  and 
nongovernmental  agencies — repositories  of  information  that  is  not  well  represented  in  the 
government.  Processes  and  tools  must  be  flexible  to  accommodate  each  nation’s  internal 
ONA  methods,  driving  the  requirements  for  procedures  and  policies  to  support  coalition 
collaboration  and  information  sharing  at  varying  levels. 
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(8)  Finding  8  ^  Participants  in  multinational  collaboration 

require  specific  training  and  experience  to  make  the  process  effective. 

Participants  must  be  well  prepared  through  education,  training,  and  experience  to 
deal  with  the  complexity  and  diversity  of  issues  that  typify  the  ONA  process  and  the  effects- 
based  approach  to  operations  that  it  supports.  In  addition,  collaboration  session  leadership 
and  moderating  require  specific  skills  and  training.  Players  emphasized  the  importance  of 
face-to-face  meetings  among  participants  in  helping  to  build  personal  confidence  and  trust. 


B.  OBJECTIVE  2  DISCUSSION 

Identify  and  assess  issues  associated  with  collaboration  and  information  sharing 
across  different  security  domains. 

1.  Objective  2  Overall  Assessment  Results 

MN  LOE  II  identified  collaboration  and  information- sharing  issues  among 
multinational  partners.  However,  the  experiment  did  not  provide  the  structure  needed  to 
assess  the  efficacy  of  the  CISP3  process  and  the  new  security  domain  architecture.  Thus, 
the  consensus  among  all  participants  and  experiment  analysts  is  that  this  objective  fell 
short  of  its  goal. 

The  reliance  on  an  unclassified  data  repository  did  not  force  a  hard  look  at  what 
can  and  cannot  be  released,  or  at  the  reasons  behind  those  determinations.  Information¬ 
sharing  issues  were  subject  to,  and  unduly  influenced  by,  current  information- sharing  rules 
that  are  too  general  and  encompassing  in  their  restrictions.  The  applicable  rules  were  not 
adequately  distinguished  by  the  context  and  the  need  for  secrecy.  The  fear  and  perceived 
risk  that  motivates  this  approach  to  information  sharing  must  be  alleviated  through  the 
realization  that  fully  effective  collaboration  requires  a  change  in  philosophy  and  policies. 
Future  experimentation  faces  a  challenge  to  use  real  data  and  issues.  Furthermore,  the 
eventual  real-world  implementation  of  a  multilevel  security  (MLS)  network  and  database 
is  needed  to  support  any  MNIS  concept  of  the  future.  The  MNIS  concept,  as  played  in 
this  event,  assumed  the  presence  of  a  MLS  network  and  database. 

The  experiment  provided  a  starting  point  for  further  development  of  solutions  to 
the  information- sharing  problem.  It  produced  evidence  to  support  the  possibility  of  strong 
international  collaboration  using  the  ONA  process,  given  appropriate  tools  and 
information-sharing  procedures  and  policies.  If  nations  can  collaborate  and  share 
information  during  peacetime,  improved  situation  awareness  may  result,  leading  to 
superior  decision-making.  Nations  should  sustain  an  open  dialogue  to  identify 
information-sharing  impediments  that  restrict  collaboration. 

Participants  observed  that  the  CISP3  and  the  future  information- sharing  process 
did  not  consider  the  various  non-U. S.  national  policies  and  procedures  for  the  release  of 
classified  infonnation  in  a  multinational  environment.  Partner  nations  did  not  portray 
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their  own  national  policies,  and  so  the  resulting  procedures,  when  applied  by  non-U. S. 
players  within  the  experiment,  were  artificial  and  constrained. 

2.  COI  2.1  Discussion  -  Does  collaboration  with  coalition  partners  across 

security  domains  improve  the  ONA? 

a)  COI  2. 1  Assessment  Results 

Collaboration  with 
coalition  partners  across 
security  domains  poten¬ 
tially  could  improve  the 
ONA.  In  this  experiment, 
the  new  cross-domain, 
information-sharing  con¬ 
cept  was  not  sufficiently 
tested  or  assessed.  Instead, 
the  experiment  focused  on 
the  care  and  maintenance 
of  the  ONA  database  rather 
than  on  the  exploitation  of 
the  information  available. 

The  experiment  became 
driven  more  by  the  nodes 
and  links  than  by  the 
investigation  of  ONA’s 
benefits  and  potential. 

Security  domains  were 
not  stressed  or  sufficiently  enforced  to  address  the  potential  situations  and  implications. 
Most  of  the  discussions  during  the  first  and  second  weeks  occurred  in  the  coalition  space. 
Figure  14  depicts  the  number  of  visits  to,  or  use  of,  each  shared  space,  including  quick 
visit-and-depart  activities. 

In  theory,  collaboration  across  security  domains  should  provide  a  more  robust, 
enhanced  situational  awareness.  In  this  LOE,  no  statistical  difference  existed  in 
participants’  self-assessed  situational  awareness  within  the  first  two  weeks,  preventing 
meaningful  comparisons  about  the  effect  of  the  cross-domain  concept. 
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Figure  14.  Spaces  Used  during  MN  LOE  11 


b)  COI  2.1  Findings 

(1)  Finding  1  ->  Due  to  a  variety  of  experimental  factors,  future 

MNIS  was  not  exercised  sufficiently. 

When  asked  if  nations  shared  information  satisfactorily  to  develop  a  multinational 
ONA,  SCDs  commented  that  the  exercise  did  not  create  conditions  that  sufficiently  tested 
information  sharing.  More  experimentation  is  required  to  fully  answer  this  question. 
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Even  though  the  working  relationships  improved  over  time,  SCDs  said,  “We  don't  know 
the  answer  to  this  question.  The  injects  didn't  seem  to  be  pertinent  or  valuable  enough  for 
people  to  absolutely  need  them  to  make  good  decisions.  We  also  didn’t  form  the 
circumstances  in  which  people  in  coalitions  had  vested  personal  or  national  interests — the 
two  key  variables  for  precluding  or  sharing  information.  Suggest  we  derive  ways  to  make 
certain  we  are  perturbing  these  two  variables  for  the  next  LOE.” 

Asked  if  the  future  MNIS  was  an  improvement  over  current  methods,  SCDs  said 
that  the  difference  was  unclear:  “[It]  appears  more  likely  that  the  learning  curve,  increased 
familiarity  with  other  players,  and  the  transition  to  a  'matrix'  collaboration  process  [versus] 
traditional  staffing  had  far  more  effect  on  the  sharing  outcomes.”  SCDs  also  noted  that 
relationships  among  players  had  been  built  up,  resulting  in  increased  trust — a  “critical  factor 
in  the  improvement  in  infonnation  sharing,”  more  so  than  the  new  MNIS  method. 

An  underlying  problem  was  that  the  experiment  did  not  use  information  sharing  as 
originally  planned.  During  Week  1,  observers  noted  that  participants  truncated  national 
disclosure  procedures,  resulting  in  no  practical  difference  between  the  current  information 
sharing  procedures  and  those  of  future  information  sharing.  In  addition,  the  injects  that 
participants  received  at  the  beginning  of  each  vignette  were  created  before  the  CISP3 
convened,  so  that  CISP3  decisions  did  not  affect  many  of  the  disclosures.  Further,  to 
reduce  the  administrative  burden  on  SoSAs  and  planners,  all  nations’  panels  decided  to  list 
exceptions  to  automatic  release  rather  than  to  develop  explicit  items  for  automatic  release. 
This  may  have  influenced  the  employment  of  the  future  information  sharing  to  reflect  that 
of  current  information  sharing. 


Finally,  insuf¬ 
ficient  time  was  allowed 
to  complete  the 
experiment’s  ambitious 
agenda.  Non-real-time 
injects  hampered  the 
foreign  disclosure 
officers’  (FDO)  work. 

The  recommended  solu¬ 
tion  was  to  extend  time 
for  the  FDO  to  work 
through  the  releasability 
process  and  for  each 
nation  to  digest  the 
newly  released  infor¬ 
mation.  Additionally,  competing  experiment  objectives,  such  as  conducting  basic 

collaboration  among  distributed  nations,  took  precedence  over  the  more  advanced  goals  of 
information  sharing. 


Strongly  Disagree 


Figure  15.  Information  sharing  across  security  domains  enhanced  the 
quality  of  information  in  the  ONA  knowledge  base. 
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(2)  Finding  2  ->  Information  sharing  across  security  domains 

enhanced  the  ONA  quality. 

Regarding  information  sharing,  positive  comments  from  respondents  indicated  that 
“something  is  always  better  than  nothing.”  When  an  information-sharing  database  is 
implemented,  participants  will  work  to  improve  it  continually.  Specifically,  interactions 
must  be  managed  to  foster  collaboration.  A  Canadian  participant  observed,  “For  the 
purposes  of  LOE,  this  is  an  effective  test.  It  worked,  and  info  was  often  used  for  the 
construction  and  selection  of  nodes,  effects,  actions,  and  resources.”  Even  though  participants 
indicated  that  the  ONA  knowledge  base  was  enhanced,  they  also  indicated  that  the  MNIS  was 
not  sufficiently  tested.  (See  Figure  15.)  Information  was  not  shared  across  real-world  security 
domains  in  this  experiment;  all  information  was  notional,  unclassified,  and  for  official  use  only. 

A  majority  of 
the  players  indicated 
that  needed  informa¬ 
tion  was  available  in 
the  ONA  knowledge 
base  during  both 
weeks  of  the  experi¬ 
ment.  (See  Figure 
16.)  Players  indicated 
that  the  database  was 
sufficient  within  the 
context  of  the  experi¬ 
ment.  However,  the 
database  still  needs 
substantial  work  before 
it  becomes  viable  as  a 

real-life  decision-making  tool,  as  this  Canadian  participant  observed: 

“The  nodes  were  fairly  comprehensive  for  the  two  countries.  Effects  were 
the  next  most  comprehensive  component.  The  actions  and  resources  were 
poorly  populated  (qualitatively  and  quantitatively),  in  the  following  senses: 

Actions  list  was  lacking  some  actions,  which  would  have  been  blatantly 
obvious  (for  example,  ministerial-level  engagement,  MNC  engagement). 
Resources  list  was  heavily  populated  with  military  assets,  which  in  Week 
l’s  scenarios  were  not  useful.  The  focus  this  week  was  on  pre-crisis,  no 
hostilities  situations,  and  that  80  percent  of  the  resources  seemed  to  be  hard 
military  assets  seemed  inappropriate.  In  this  week’s  situations,  resources 
such  as  multinational  councils,  individuals  within  other  partners’  countries, 
and  a  variety  of  governmental  officials  were  found  to  be  the  most  useful 
resources  for  the  ENAR  linkages  that  were  discussed.  These  resources  were 
also  added  as  the  need  arose.  There  is  an  important  need  for  nontraditional 
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Figure  16.  The  ONA  knowledge  base  contained  the  information  needed. 
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resources  to  be  included  in  this  list  if  we  are  truly  to  consider  diplomatic,  and 
nonhard  military  approaches  to  these  situations.” 


Most  significantly,  no  statistical  difference  existed  from  Week  1  to  Week  2  in  the 
players’  perceptions  of  the  degree  to  which  information-sharing  procedures  allowed  the 
sharing  of  information  during  the  ONA  development  process.  The  capability  to 
communicate  instantly  with  other  countries  and  to  collaborate  from  a  common  database  is 
invaluable.  However,  the  information-sharing  procedures  used  within  the  experiment 
were  not  effective,  according  to  participants’  perceptions.  Many  commented  that  the 
large,  cumbersome  information-exchange  requirements  list  created  confusion  and  took  an 
exorbitant  amount  of  time  to  get  through. 

(3)  Finding  3  ->  Smaller  groups  were  more  conducive  to 
collaboration  than  were  the  larger  groups. 

Collaboration  among  nations  across  the  information-sharing  spaces  led  to  discussions 
about  sharing  information  more  efficiently.  As  an  attempt  to  increase  multinational 
collaboration  within  the  constraints  of  the  collaborative  technology,  the  coalition  agreed  to 
break  into  small  groups  organized  by  PMESII  (analysts’)  and  DIME  (planners’)  areas  toward 
the  end  of  the  second  week. 

Nations  were  assigned  to 
moderate  these  group  efforts  to 
link  effects,  nodes,  actions, 
and  re-sources.  These  small 
groups  fostered  better  co¬ 
llaboration,  better  information 
sharing,  and  increased  partici¬ 
pation  among  the  coalition 
nations.  Regarding  facilita¬ 
tion,  some  nations  preferred 
full  consensus  to  majority  rule. 

As  depicted  in  Figure 
17,  discussion  tool  usage 
increased  on  the  days  that 
smaller  groups  were  used.  The 
use  of  Internet  Protocol 
phones  caused  one  spike  on 
February  24. 

(4)  Finding  4  ->  No  significant  statistical  difference  in  situational 
awareness  occurred  from  Week  1  to  Week  2  of  the  experiment. 

As  one  of  its  objectives,  the  experiment  evaluated  the  effect  of  the  information¬ 
sharing  mechanisms  on  situational  awareness,  using  a  comparison  of  Weeks  1  and  2. 


Figure  1 7.  Discussion  Tool  Usage  during  MN  LOE II 


47 


Figure  18  shows  the  total  probe 
performance  for  Week  1,  when 
traditional  information  sharing 
procedures  were  used,  and  for 
Week  2,  when  the  new  CISP3 
“future”  information  sharing 
mechanism  was  applied. 

The  data  suggests  no  sig¬ 
nificant  difference  in  the  players’ 
abilities  to  answer  the  probe 
statements  in  Week  1  and  Week  2, 
probably  because  of  similar 
information-sharing  mechanisms 
used  during  the  period.  As  for 
information  sharing  itself,  during 
Week  2,  players  more  rigorously 
scrutinized  intelligence  products 
and  therefore  reduced  the  amount 
of  infonnation  disclosed.  This 
unplanned,  unanticipated  part  of 
the  experiment  developed  as  each 
nation  made  increasingly  realistic 
information  disclosures  over  time. 

As  shown  in  Figure  19, 
probe  questions  included  inject- 
and  background-based  material, 
with  a  marginal  difference  in  the 
interaction  between  type  of  probe — 
“distracters”  and  “injects” — and 
information-sharing  mechanism. 
The  data  suggests  that  the  slight, 
statistically  insignificant  deterior¬ 
ation  in  probe  performance  from 
Week  1  to  Week  2  is  due  not  to 
inject-based  material,  but  to  back¬ 
ground  infonnation,  or  distracters. 
Fewer  correct  answers  were  based 
on  background  material,  possibly 
because  players  did  not  read  or 
pay  as  much  attention  to  the 
background  material  in  Week  2  as 
they  did  in  Week  1.  Players 
commented  that  the  background 
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Figure  18.  Probe  Performance  for  Week  1 
(Current  Information  Sharing) 
and  Week  2  (Future  Information  Sharing) 
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Figure  19.  Probe  Performance  by  Weeks  1  and  2 
and  by  Type  of  Probe  Questions 
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Figure  20.  Probe  Performance  by  Role  of  Player 
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material  to  the  vignettes  was  relevant  only  for  answering  the  probe  questions  and  not  for 
play,  and  consequently,  they  stopped  bothering  with  it. 

Probe  Performance  and  Role 


As  depicted  in  Figure  20,  probe  performance  by  role  of  player  was  examined  to 
determine  the  difference  in  situational  awareness.  Al-though  no  difference  was  observed 
from  one  week  to  the  next,  differences  within  each  week  were  found  between  planners  and 
SoSAs.  The  data  suggests  that  the  analysts  were  better  able  to  answer  the  probe 
statements  than  the  planners  were,  probably  because  they  knew  more  about  the  inject 
material.  Planners  may  have  had  a  better  general  overview  of  the  injects  and  of  the 
general  situation,  but  were  less  able  to  discern  the  fine  detail  required  in  the  probes. 


Self-Assessed 
Situational  Awareness 

Participants’  self- as¬ 

sessed  situational  awareness 
complemented  the  actual 
probe  performance,  showing 
no  significant  statistical  dif¬ 
ference  from  the  beginning  to 
the  end  of  the  experiment. 
(See  Figures  21  and  22.)  Sit¬ 
uational  awareness  appeared 
to  be  average  across  the 
board,  although  the  large 
amount  of  data  was  difficult  to 
maintain.  Participants’  famili¬ 
arity  with  the  tools  and  their 
functionality  also  affected  their 
assessments:  “The  way  you 

have  to  navigate  around  Groove 
and  the  undisciplined  manner 
we  are  moving  and  sharing  the 
info  make  it  hard  to  be  comfort¬ 
able  with  what  you  should 
know.”  To  familiarize  them 
with  the  probe  process,  partici¬ 
pants  answered  questions  during 
Week  0  about  situational 
awareness  for  the  overall  ex¬ 
periment  construct.  Figure  22 
presents  that  data  as  a  baseline 
for  the  experiment  weeks. 


Figure  21.  Mean  Rating  of  Overall  Situational  Awareness 


Figure  22.  Players  Perceived  SA  Week  0  through  2 
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Figure  23.  Depiction  of  the  ONA  Process 


(5)  Finding  5  ->  Releasability  violations  decreased  as  familiarity  with 

the  functionality  of  the  tools  in  the  collaborative  environment  increased. 

This  area  of  assessment  was  a  direct  result  of  the  artificiality  built  into  the 
experiment.  Participants  were  directed  to  report  all  violations,  all  of  which  resulted  from 
unfamiliarity  with  the  tools  and  the  experiment  construct.  Reported  releasability 
violations  decreased  from  Week  1  to  Week  2.  Nineteen  distinct  violations  were  reported 
during  Week  1,  plus  22  duplicate  reports.  Eight  violations  were  reported  during  Week  2. 
Based  on  players’  and  observers’  comments,  this  significant  decrease  was  due  more  to  an 
increased  comfort  level  with  the  tools  than  with  the  differences  in  infonnation-sharing 
methods.  In  the  future,  a  more  structured  training  regime  should  be  used  before  a  similar 
database  is  used. 

The  releasability  problems  encountered  were  related  to  human  error  and  to  the 
familiarity  or  comfort  level  in  using  the  tools.  Most  inadvertent  releases  resulted  from  a 
participant’s  misunderstanding  of  sharing  rules  or  from  incorrect  assumptions  about  the 
“buttonology”  in  the  ONA  database.  Information-sharing  violations  will  occur  in  the  real 
world,  as  well,  due  to  operator  error,  technical  glitches,  and  operator  training  deficiencies. 

(6)  Finding  6  More  in-depth  collaboration  resulted  from  famil¬ 

iarity  with  the  tools  and  fellow  participants;  from  the  increased  use  of 
moderators;  from  organization  into  smaller,  structured  groups;  and  from  the 
refinement  of  business  rules. 

Players,  controllers,  and  observers  alike  commented  that  the  use  of  moderators  or 
facilitators  improved  the  process.  Senior  concept  developers  also  addressed  this  issue  during 
the  azimuth  and  in-focus  sessions.  A  more  structured  leadership  style  is  needed  within  a 
collaborative  enviromnent  to  organize  efforts  and  to  focus  on  objectives.  Refined  business 
rules  also  provided  structure  to  the  collaborative  process. 
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Throughout  the  experiment,  players  collaborated  more  as  they  became  more 
familiar  with  Groove  capabilities,  with  ONA  database  tools,  and  with  each  other  in  this 
“virtual”  environment,  which  increased  their  comfort  levels.  Senior  concept  developers 
recommended  that  the  leaders  personally  meet  each  other  and  work  together  during  the 
pre-experiment  workups. 

In  comments  echoed  by  each  participating  nation,  an  American  controller  summed 
up  the  effect  of  smaller  groups  on  the  collaboration  process:  “Small  groups  in  functional 
disciplines  (all  'P'  of  PMESII  together)  collaborated  better  than  large  gaggles  where  most 
people  remained  silent  on  the  line  ...  If  the  goal  is  to  produce  maximum  collaboration,  then 
do  it  via  smaller  [multinational]  teams.  If  you  are  in  the  crisis  mode,  do  it  via 
[multinational]  teams  only  to  build  the  consensus  you  need  for  [multinational]  operations.” 


3.  COI  2.2  Discussion  -  What  impact  does  information  sharing  have  on 
each  phase  of  the  ONA  process? 

a)  COI  2.2  Assessment  Results 

MN  LOE  II  was  conducted  as  a  series  of  related  events — three  workshops 
followed  by  the  time-constrained  experiment.  The  workshops  were  used  to  train  players 
in  the  ONA  process  and  to  conduct  the  earlier  phases  of  the  process — red-and-blue  views 
and  wargaming.  The  experiment  focused  on  the  later  phases — effects,  nodes,  actions, 
resources,  and  their  linkages — but  insufficient  time  remained  to  play  the  last  phase, 
second-and-third-order  effects. 

Infonnation  sharing  had  little  effect  on  any  of  the  phases  of  the  ONA  process.  By 
design,  infonnation  sharing  was  not  attempted  during  the  workshops.  But  even  during  the 
experiment,  infonnation  sharing  was  rarely  emphasized,  since  an  under-developed  CISP3 
concept  meant  that  current  and  future  procedures  were  similar.  The  last  two  phases  received 
little  experiment  emphasis  or  were  not  played  at  all.  Figure  23  depicts  the  ONA  process. 

b)  COI  2.2  Findings 

(1)  Finding  1  Information  sharing  did  not  affect  the 

development  of  red  and  blue  views  or  of  wargaming  because  they  were  not 
played  during  the  experiment. 

The  red/blue  views  and  wargaming  phases  were  conducted  during  the  workshops, 
where  players  learned  the  ONA  process.  The  workshops  did  not  employ  the  collaborative 
tool,  and  discussions  about  red/bluc  views  and  wargaming  occurred  using  presentations  in 
meeting  rooms,  intentionally  without  the  experiment's  infonnation-sharing  mechanisms.  To 
save  time,  these  phases  would  not  be  revisited  after  the  workshops. 
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Some  players’  and  controllers’  survey 
comments  indicated  that  the  red/blue  view  was  not 
played  during  the  experiment.  Players  said  that  they 
thought  that  the  red/blue  view  and  wargaming 
processes  never  were  followed  and  that  no 
discussion  of  red/blue  capabilities,  vulnerabilities, 
and  intentions  occurred.  Some  players  did  not  know 
if  they  were  supposed  to  be  wargaming.  Players 
also  said  that  information-sharing  procedures  were 
not  implemented,  and  that  time  did  not  allow  real- 
world  strategic  planning  with  long-term  impacts. 
They  did  not  get  into  the  wargaming  process  enough 
to  “draw  out”  national  perspectives.  All  information 
was  unrestricted  and  from  open  sources  during 
development  of  these  phases.  Players  held  no 
“nonreleasable”  information  that  would  have  altered 
coalition  views  as  developed  in  this  exercise. 

Certain  groups  of  players  emphasized  that 
wargaming  was  a  critical  part  of  the  ONA  process, 
when  issues  related  to  other  phases  could  be 
discussed.  Subsequently,  the  U.S.  planner  pointedly 
incorporated  wargaming  into  the  business  rules  for 
ONA  procedures.  In  reality,  wargaming  as  a 
deliberation  process  consumed  too  much  experiment 
time  and  had  to  be  curtailed  in  order  to  pursue 
development  of  procedures  for  effects,  actions,  and 
resources — the  experiment’s  designed  focus. 


Frequency  with  which  f  litre  irformation  sharing 
enhanced  the  range  of  effects  available  to  the  coalition 


Figure  24.  Information  sharing  did  not 
influence  the  effects  phase. 


(2)  Finding  2  Future 

information  sharing  had  no  influence 
on  the  effects,  nodes,  actions,  and 
resources  phases  of  the  ONA  process. 

Because  of  time  constraints,  the  experiment 
focused  on  the  development  of  effects,  nodes, 
actions,  and  their  linkages.  However,  information 
sharing  rarely  affected  these  phases.  When  players 
and  controllers  were  asked  how  frequently  future 
information  sharing  enhanced  the  range  of  options 
available  to  the  coalition  for  a  particular  phase,  the 
predominant  response  was  “almost  never.”  (See 
Figures  24,  25,  and  26.)  Survey  comments 
consistently  made  these  points: 


Figure  25.  Information  sharing  did  not 
influence  effect-to-node  linkages. 
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Figure  26.  Information  sharing  did  not 
affect  the  Action  Stage. 


□  Information-sharing  rules  based  on  the  CISP3  concept  need  more  refinement 
and  experimentation. 

□  Infonnation  sharing  rarely  had  sufficient  emphasis  to  affect  these  phases. 

□  No  significant  change  occurred  from  current  to  future  infonnation  sharing. 

(3)  Finding  3  Information  sharing  did  not  affect  resources  or 

second-  and  third-order  effects  during  MN  LOE  II. 

Since  resources  and  second-  and  third-order  effects  were  expected  to  receive 
minimal  play  during  the  experiment,  experiment  designers  did  not  emphasize  them.  Actions 
had  priority  consideration  over  generic,  high-level  resources.  The  ONA  process  did  not 
progress  to  second-  and  third-order  effects  during  the  experiment. 

Resources  were  debated  more  than  expected  during  player  collaboration,  but 
information  sharing  was  not  included.  Survey  results  revealed  that  sources  of  disagreement 
among  nations  included  nodes,  actions,  and  resources.  Players  did  not  mention  infonnation 
sharing  in  connection  with  resource  development  because  it  was  not  stressed  in  the 
experiment  or  because  they  did  not  recognize  its  relevance. 

As  predicted,  second-  and  third-order  effects  were  rarely  addressed  during  the 
experiment,  and  so  the  influence  of  infonnation  sharing  was  not  observed. 


4.  COI  2.3  Discussion  -  Does  the  MNIS  future  operational  concept 
accelerate  the  ONA  process? 

a)  COI  2. 3  Assessment  Results 

Since  the  CISP3  process  was  not  adequately  defined  nor  implemented  to  protect 
classified  material,  the  MNIS  concept  was  tested  only  marginally  in  this  experiment.  The 
ONA  process  was  not  proven  to  be  faster;  more  empirical  testing  is  needed.  Continued 
efforts  must  occur  to  further  develop  the  MNIS  concept,  whose  inherent  value  lies  in  its 
potential  to  improve  the  speed  at  which  the  ONA  database  is  built.  Future  technological 
advances  and  synchronization  tools  may  provide  essential  support  to  this  process. 

In  principle,  the  involvement  of  several  nations  and  the  subsequent  coordination 
required  may  slow  the  decision-making  process.  In  peacetime,  the  quality  of  the  results 
increases  when  more  partners  are  involved.  But  in  a  crisis,  stringent  time  limits  determine  the 
required  speed  of  decision-making.  The  possibility  of  real-time  sharing  of  infonnation  offers  a 
substantial  advantage.  Such  an  exchange  will  accelerate  greatly  the  development  of  a  common 
knowledge  base,  and  at  the  same  time,  will  enhance  the  quality  of  the  infonnation  shared. 
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b)  COI  2.3  Findings 


(1)  Finding  1  ->  MNIS  supported  the  construction  of  an  ONA 

database,  but  no  difference  was  determined  between  MNIS  “as  is”  and 
“future”  processes. 

Additional  CISP3  information  exchange  requirement  categories  to  cover  blue  force 
information,  as  well  as  an  overall  streamlined  infonnation  exchange  requirement  list,  may 
speed  parts  of  the  ONA  process.  With  the  additional  training  benefit  gained  from  the 
work  performed  during  the  first  week,  the  FDOs  handled  only  a  couple  of  injects  at  a  time 
and  responded  quickly  to  requests  to  release  information.  But  the  need  to  collaborate 
with  multiple  nations  actually  may  lengthen  the  time  required  to  build  the  ONA,  whereas  a 
U.S.-only  ONA  could  be  built  more  quickly. 


Additional  observations 
from  all  nations  showed  no  discer- 
nable  difference  in  the  information 
sharing  and  collaboration.  Further¬ 
more,  no  statistical  difference 
existed  among  effects,  actions,  and 
resources  data  records  available  in 
the  ONA  knowledge  base.  As 
Figure  27  shows,  equal  numbers  of 
effects,  actions,  and  resources  were 
created  and  made  available  to 
participants  throughout  the  entire 
experiment. 


02/17/03  02/18/03  02/19/03  02/20/03  02/24/03  02/25/03  02/26/03  02/27/03 


Figure  27.  Effects-Actions-Resources  Created  versus  Time 


Similarly,  the  number  of  node 
data  records  available  in  the  ONA 
knowledge  base  did  not  significantly 
differ  from  Week  1  to  Week  2.  (See 
Figure  28.)  The  slight  change  seen 
during  the  last  few  days  of  Week  2 
may  have  been  due  to  experiment  de¬ 
sign  changes,  such  as  working  within 
DIME  or  PMESII  groups,  rather  than 
to  information-sharing  methods. 

The  number  of  node  data 

records  available  directly  reflects  the 

spaces  that  a  country  may  visit.  All 

member  nations  were  members  of  the 

,  ■  ■  ,  .  Figure  28.  Number  of  Node  Data  Records  Available 

coalition  spaces  and  the  private 

spaces.  Germany  and  Canada  were  members  of  the  multilateral  and  trilateral  spaces;  the 
United  States  and  the  United  Kingdom  were  members  of  the  multilateral  and  bilateral  1 
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spaces.  More  node  data  records  were  available  in  the  trilateral  space  than  in  the  bilateral  1 
space.  A  similar  comparison  may  be  made  between  Australia  and  the  regional  cell. 

(2)  Finding  2  ->  No  conclusive  test  determined  that  MNIS 

accelerated  the  ONA  process.  MNIS  concepts  need  further  development. 

During  the  final  hot  wash  participants  within  the  regional  cell  reflected  that  they 
could  not  determine  whether  the  process  had  accelerated.  The  regional  cell  conducted 
multinational  information  sharing,  but  when  the  requirements  for  automatic  release  were 
not  clearly  met,  review  and  discussion  reverted  to  normal  procedures. 

The  Canadians’  final  hot-wash  comments  concluded  that  the  ability  to  move 
quickly  through  the  ONA  process  was  not  determined  exclusively  by  information-sharing 
rules;  information  availability  and  analysis  time  constraints  also  were  factors. 

Observers,  participants,  and  a  SCD  suggested  that  the  process  could  be  accelerated 
through  information  sharing  and  direct  communication.  The  more  subjective  the 
information,  the  more  directly  it  must  be  communicated. 

As  depicted  in  Figure  29 
Week  2’s  information-sharing 
procedures  did  not  accelerate 
the  ONA  process: 

□  “The  only  speed  en¬ 

hancements  were  as  a 
result  of  changes  in 
experimental  proced¬ 
ures,  players’  familiarity 
with  the  tools,  and 
anticipation  of 

ENDEX.” 

□  “Did  not  see  that  there 
were  really  any  differ¬ 
ences  in  the  information¬ 
sharing  process  from 
Week  1  to  Week  2  (although  there  [were]  supposed  to  be).” 

□  “Still  worked  in  three  spaces  (COA,  ML,  TL),  due  to  injects  coming  in  to  each  of  them.” 

□  “It  may  have  [accelerated  the  ONA  process]  in  the  real  world,  but  in  the 
experiment  environment,  there  was  too  much  that  we  did  not  replicate;  for 
example,  national  FDOs  were  considered  the  originating  authority  for  all 
messages.  Using  the  white  cell  would  have  been  a  better  choice.” 

□  “Here's  an  example  of  the  'effectiveness'  of  CISP3.  Four  injects  were  received  for 
vignette  #4,  one  of  which  had  a  red  banner  at  the  top  advertising  that  it  was  not 
releasable  to  the  COA  without  going  through  the  CISP3  process;  the  other  three 


,  most  players,  observers,  and  controllers  agreed  that 


Figure  29. Did  Week  2’s  information-sharing  procedures 
accelerate  the  ONA  process? 
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had  no  such  banner.  However,  only  one  of  the  four  could  be  released  without 
going  through  the  CISP3  process.” 

□  “But  it  was  entirely  artificial.  We  could  make  it  as  fast  as  we  wanted.” 

□  “It  took  more  time  for  discussion  of  the  issues  that  ‘failed’  the  automatic  release 
test.  Those  that  passed  through  the  screen  automatically  probably  would  have 
been  passed  along  quickly  without  the  use  of  the  matrix.” 

□  “No,  but  level  of  quality  was  increasing.” 

□  “No.  Did  not  observe  any  appreciable  difference  in  the  ONA  process  between 
Weeks  1  and  2  [that  was]  attributable  to  information  sharing.” 

□  “CISP3  was  unworkable.” 


5.  COI  2.4  Discussion  -  Can  information  releasability  procedures  keep 
the  ONA  data  current? 

a)  COI  2. 4  Assessment  Results 

Most  participants  agreed  that  infonnation  release  procedures  implemented  in  this 
experiment  kept  the  ONA  data  current.  However,  they  also  noted  that  those  procedures 
would  be  extremely  difficult  to  implement  in  the  real  world.  Due  to  the  small  amount  of 
data  that  needed  to  be  updated  in  the  database  during  the  course  of  the  experiment, 
participants  cautioned  against  concluding  that  these  procedures  would  be  effective  in  an 
operational  environment. 

b)  COI  2.4  Findings 

(1)  Finding  1  ->  Information  release  procedures  potentially  may 

affect  the  currency  of  information  needed  to  conduct  a  successful 
multinational  ONA. 

In  the  real  world,  keeping  the  database  current  is  a  challenge,  and  release 
procedures  would  definitely  have  an  impact  on  the  quality  of  the  database. 

As  Figure  30  depicts,  players  found  minimal  correlation  in  this  experiment  between 
releasability  procedures  and  the  currency  of  the  knowledge  base.  Each  vignette  lasted  only 
two  days,  and  the  few  information  injects,  provided  at  the  beginning,  allowed  the  ONA 
database  to  be  updated  easily.  Even  so,  updates  were  not  always  posted  as  soon  as  they  were 
made  available  and  so  were  a  secondary  consideration.  After  those  injects  were  shared,  no 
basis  existed  to  detennine  the  currency  of  the  database. 
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(2) 

Finding  2  -> 

Participants  identi¬ 
fied  issues  that  in¬ 
hibited  their  abili¬ 
ty  to  keep  the  data¬ 
base  current. 

Release  procedures 
potentially  may  inhibit 
keeping  the  ONA  current,  but 
they  are  not  going  to  “go 
away.”  In  addition,  national 
policies  often  inhibit  release 
of  information  and  database 
currency.  Because  the  CISP3 
was  not  exercised  extensively 
during  Week  2,  its  potential  to 
facilitate  the  currency  of  the  database  was  not  assessed. 


Information  releasabilty  procedures  kept  the  ONA  knowledge  base  current. 


Strongly  Disagree 


Figure  30.  Perception  of  Impact  of  Information-Sharing  Procedures 
on  Currency  of  ONA 


(3)  Finding  3  ->  Participants  identified  release  procedures  that 

facilitated  their  ability  to  keep  the  database  current. 

U.S.  participants  who  were  familiar  with  the  FDO  process  benefited  from  the  FDO 
discussion  of  releasability  issues.  Intuitively,  fewer  release  restrictions  meant  a  more 
effective  flow  of  information.  Yet,  business  rules,  specific  information-sharing  domains, 
and  releasability  procedures  were  useful.  Multilevel  releases  allowed  information  to  reach 
a  wider  audience,  even  if  not  the  full  coalition  area.  In  addition,  the  process  was 
facilitated  by  prompt  action,  discussion,  and  interaction  between  the  players  and  CISP3  to 
resolve  issues,  as  well  as  by  national  foreign  disclosure  procedures. 


C.  SPIN-OFF  FINDINGS 


A  result  of  the  discovery  nature  of  the  experiment  and  the  data  collection  process, 
these  analytical  findings  are  based  outside  the  realm  of  the  examined  objectives  and  COIs. 

1.  Spin-Off  Findings  Assessment  Results 


Finding  1  ->  At  times  conducted  asynchronously,  the  continuous  ONA  process 
should  be  performed  with  minimum  distinction  between  the  methods  and 
release/disclosure  policies  that  are  used  during  and  before  a  crisis  situation. 

Built  on  a  knowledge  base,  ONA  is  a  continuous  process  that  requires  continuous 
collaboration,  which  allows  leaders  to  make  good,  fast,  effective  decisions.  Using 
analysts’  expertise,  the  value  of  existing  knowledge  is  based  on  its  relevancy  to  the 
situation.  In  addition,  analysts  “create”  knowledge  through  a  lengthy  process  of  data 
collection  and  effective  collaboration.  Differing  policies  for  crisis  and  pre-crisis  release  or 
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disclosure  may  impede  timely  access  to  critical  information.  Effective  ONA  depends  on 
successful  information  sharing  in  a  pre-crisis  situation. 

Finding  2  ->  Partner  nations  strongly  favored  political  agreements  as  well  as  a  legal 

basis  to  establish  and  conduct  the  collaborative  ONA  information-sharing  process. 

Partner  militaries  must  operate  under  political  and  legal  controls  that  differ  from  those 
of  the  U.S.  These  approved  agreements  would  give  shape  and  legitimacy  to  the  ONA  process. 

Finding  3  ->  ONA  database  capabilities  must  be  improved  to  support  the 

ONA  process. 

Participants’  comments 
and  observations  indicated  that, 
within  the  context  of  this  limited- 
objective  experiment,  the  ONA 
database  adequately  supported 
the  experiment  objectives. 

Furthermore,  as  Figure  31  shows, 
a  significant  number  of  respond¬ 
ents  indicated  that  the  tools  used 
allowed  easy  search  and  access, 
in  spite  of  some  limitations. 

Participants  indicated  that  im¬ 
provements  to  the  ONA  data¬ 
base  are  needed  in  order  for  it  to  Figure  31.  Users  indicate  that  tools  allowed  easy  search  and 

support  adequately  a  real-world  access- 

multinational  ONA  process.  Suggested  improvements  included: 

□  Query  response  times 

□  Advanced  database  filters,  sorting  and  search  capabilities 

□  User  selectable  tracking  and/or  notification  of  database  updates  and  changes 

□  Priority  and  hierarchy  associations  with  linkages  and  effect 

□  Easier  navigation  through  the  database  among  nodes,  effects,  and  linkages 

□  Support  for  the  constant  back-and-forth  among  nodes,  effects,  and  various 
displays  that  is  required  in  the  cognitive  process  of  ONA 

□  Tagging  of  records  with  reliability,  confidence,  and  date  of  information 

□  Simultaneous  use  of  different  parts  of  the  database  without  needing  to  open 
more  collaborative  tool  workspaces 

□  Collaborative  work  on  the  database 

□  Tools  to  assist  in  the  collaborative  data-review  process 

□  Enhanced  graphical  displays  of  data,  including  three-dimensional  visualization 

□  Creation  of  links  in  the  graphic  display 

□  Internet  access. 


Strongly  Disagree  Disagree  Agree  Strongly  Agree 
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Finding  4  ->  To  transform  information  into  shared  knowledge, 
information  sharing  must  be  assessed  and  evaluated. 

To  accomplish  this  transformation,  every  document  to  be  shared  should  contain  the 
rationale  for  information  sharing.  Since  knowledge  is  represented  in  the  ONA  database 
through  nodes  and  their  linkages  or  through  inserts  to  the  effect,  actions,  and  resources 
lists,  the  document  originator  should  include  the  effect  of  this  infonnation  on  the  database. 
This  would  accelerate  staff  processing  efforts  to  identify  the  addressee  and  would  support 
multilateral  collaboration  by  helping  the  receiver  to  assess  the  proposal.  The  originator’s 
assessment  also  should  address  the  action  needed  within  the  ONA  database  and  effect  on 
the  effect,  node,  action,  or  resource. 

Finding  5  ->  To  keep  track  of  shared  information,  a  data  documentation 
system  must  be  incorporated  into  the  collaboration  tool  used  during  the 
ONA  process. 

This  documentation  system  should  store  every  incoming  message  by  time  and 
originator.  Staff  processing  of  the  document,  like  evaluation  and  assessment,  should  be 
recorded  in  the  information  hie.  To  support  shared  knowledge,  the  document  system  also 
should  insist  that  the  team  member  add  his  assessment  and  proposed  categorization  of  this 
information. 

Finding  6  ->  The  collaboration  tool  suite  must  provide  robust  audio,  text, 
and  visualization  capabilities  to  support  a  distributed  collaboration. 

The  tool  suite  used  must  effectively  facilitate  multinational  collaboration.  The 
tools  used  for  this  experiment  had  many  shortcomings  that  inhibited  effective  coalition 
collaboration.  Most  participants  blamed  the  collaborative  tool  specifically,  while  others 
pointed  to  inadequate  bandwidth  allocated  on  the  CFBL  net. 

Participants  commented  that  some  collaboration  tool  functions  that  were  not 
available  in  this  experiment  must  be  implemented  to  support  a  real-world  multinational 
ONA  process,  including: 

□  Robust  audio  capabilities 

□  Video  teleconferencing 

□  Accommodation  of  numerous  users  in  the  same  collaborative  workspace 

□  Auditorium  function  for  presentation  of  briefings 

□  More  open  workspaces,  without  a  resulting  application  or  network  crash 

□  Broadcast  feature  within  collaborative  environment  to  pass  infonnation  to  ah 
participants,  regardless  of  workspace 

□  Audio  controller  to  allow  operator  to  open  multiple  workspaces  and  to  select 
appropriate  audio 

□  Enhanced  ability  to  track  text  chat  during  audio  chat 

□  Indication  of  the  last  message  read  in  text  chat 
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□  Shared  view  of  an  individual’s  displays  and  products 

□  View  of  names  of  all  message  recipients 

□  Shared  graphic  products,  pictures,  and  briefs 

□  Enhanced  threaded  discussions 

□  Enhanced  whiteboard/sketchpad  functions 

□  Enhanced  automated  change/update  notification  subscription  functions 

□  Online  reference  documents. 
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V. 


EXPERIMENTAL  PROCESS  LESSONS 


Many  assessments  and  lessons  learned  are  documented  and  published  as  a  result  of  the 
success  of  MN  LOE  II.  A  mixture  of  lessons  learned,  comments,  observations,  and 
recommendations  offered  healthy,  constructive  criticism  of  the  experiment.  Input  was  critically 
reviewed  in  order  to  focus  on  the  main  issues  and  to  lend  credence  to  the  rest.  Presented  in  a 
Joint  Universal  Lessons  Learned  fonnat,  the  lessons  learned  are  not  analytical  DOTMLPF 
insights  or  recommendations,  but  rather  offer  suggestions  to  improve  the  experimental 
process. 

The  main  lessons  learned  aimed  to  facilitate  the  successful  completion  of  the  next 
multinational  experiment.  After  those  lessons  are  implemented,  or  possibly  at  the  same 
time,  the  ancillary  lessons  learned  should  be  implemented  if  time  and  resources  permit. 
See  Appendix  H  for  more  information. 

Lessons  learned  are  presented  in  this  fonnat: 

Title  of  Lesson  Learned: 

□  Observation:  This  is  the  point  of  concern. 

□  Discussion:  The  discussion  explains  the  observation  and  provides  more 
background  of  the  potential  problem  or  solution. 

□  Lessons  Learned:  What  was  learned  from  the  observation  that  was  made? 

□  Recommendation:  What  is  the  recommendation  for  future  experimentation? 

1.  Title:  Better-Defined  Experiment  Objectives  and  Issues  To  Be  Explored  or  Assessed 

□  Observation:  Clearly  defined  experiment  objectives  are  crucial  to  concept 
developers  and  experiment  analysts.  Hindsight  revealed  that  some  of  the  COIs 
had  multiple  meanings. 

□  Discussion:  Sufficient  focus  is  needed  on  a  realistic  number  of  objectives, 
given  the  limited  available  experiment  time.  These  objectives  must  be 
identified,  clearly  defined,  and  articulated  early  in  the  planning  process.  The 
distinction  between  discovery  and  hypothesis  testing  must  be  considered  in  the 
experiment  design.  COIs  must  be  succinct  enough  to  allow  the  appropriate 
measures  of  effectiveness  and  performance  to  be  identified.  Implied 
objectives,  such  as  training,  must  be  clearly  addressed  to  the  experiment 
audience.  Daily  player  objectives  must  be  stated  at  the  start  of  each  day;  is 
today’s  focus  on  process  or  product?  This  problem  developed  because 
experiment  analysts  were  not  intimately  involved  with  all  stages  of  the 
planning  process,  starting  with  the  pre-concept  development  conference.  The 
core  MN  LOE  II  analysts  met  after  the  initial  planning  conference;  some  joined 
during  the  mid-planning  conference  or  later.  Plans  for  MNE  III  include 
attendance  of  experiment  analysts  at  the  concept  development  conference. 
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Experiment  analysts  will  conduct  workshop-type  meetings  in  conjunction  with 
the  planning  events. 

□  Lessons  Learned:  All  objectives  should  be  identified,  clearly  defined,  and 
articulated  early  in  the  planning  process,  with  experiment  analysts  intimately 
involved  at  every  stage  in  this  planning  process. 

□  Recommendation:  Identify,  clearly  define,  and  articulate  experiment 

objectives  for  MNE  III  now,  with  the  experiment  analysts  directly  involved. 

2.  Title:  Experiment  Planning  and  Design  Involvement 

□  Observation:  Some  of  the  experiment  concept  experts  were  not  involved  in 
experiment  planning  until  late  in  the  process;  experiment  analysts  became  fully 
involved  in  experiment  planning  during  the  mid-planning  conference.  At  that 
point,  they  could  affect  the  experiment  design  only  minimally. 

□  Discussion:  Concept  developers  from  all  of  the  concepts  to  be  evaluated  must 
be  directly  involved  early  in  the  planning  process.  They  must  define  their 
concepts  so  that  the  experiment  planners  and  analysts  may  design  the 
experiment  to  collect  the  data  needed  to  examine  them. 

Manning  limitations  and  commitments  to  other  experimentation  events 
precluded  most  experiment  analysts  from  becoming  fully  involved  in  the 
planning  process  prior  to  the  mid-planning  conference.  Experiment  analyst 
involvement  is  needed  to  ensure  that  the  experiment  design  allows  data 
collection  and  assessment  to  address  the  experimentation  objectives. 

□  Lessons  Learned:  Experiment  analysts  and  concept  developers  must  be 
intimately  involved  in  the  experiment  planning  and  design  process  at  the 
earliest  possible  opportunity. 

□  Recommendation:  Involve  experiment  analysts  and  concept  developers  in  the 
experiment  planning  and  design  process  from  the  initiation  of  the  concept 
development  conference  through  experiment  execution,  furthermore,  the 
planning  events  must  include  time  for  all  partner  nation  experiment  analysts  to 
build  assessment  plans  in  conjunction  with  the  workshops. 

3.  Title:  Changes  to  Experiment  Design  During  Execution 

□  Observation:  Changes  were  made  to  procedures,  database  interfaces,  and  use 
of  Groove  spaces  during  experiment  play. 

□  Discussion:  A  strong  effort  was  made  during  MN  LOE  II  to  coordinate  all 
changes  through  the  lead  experiment  analyst.  Pros  and  cons  were  weighed  on 
the  consequences  of  such  changes,  i.e.,  particular  shifts  would  cause  a 
paradigm  change  from  hypothesis  testing  to  a  discovery  type  of  experiment.  In 
the  long  run,  the  shift  to  discovery  was  determined  to  be  more  beneficial  in  this 
particular  LOE.  The  benefits  of  using  discovery  methods  outweighed  the 
limitations,  such  as  limits  to  the  analysts’  ability  to  compare  results  from  one 
vignette  to  another  and  from  the  first  experiment  week  to  the  second. 
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□  Lessons  Learned:  During  future  experiments,  all  changes  must  continue  to  be 
coordinated.  Before  implementing  changes,  future  experiment  managers  must 
understand  the  potential  implications  and  effects  on  experiment  analysis. 
Uncontrolled  changes  to  experiment  conduct  preclude  the  ability  to  test 
experiment  hypotheses.  If  the  experiment  goal  is  discovery — not  hypothesis 
testing — then  greater  latitude  exists  for  making  changes. 

□  Recommendation:  Experiment  managers  should  continue  to  coordinate 
proposed  changes  in  experiment  design  with  the  experiment  analysts  and 
partner  nations  before  allowing  such  changes  to  be  implemented.  Affected 
participants  may  need  to  be  notified  of  those  changes  when  the  decision  is 
made  to  implement  them. 

4.  Title:  Time  Constraints 

□  Observation:  Insufficient  time  was  allowed  to  complete  training  and  to  explore 
key  issues  of  the  experiment. 

□  Discussion:  Many  problems  stemmed  from  lack  of  time  or  misused  time.  It  is 
impractical  for  experiments  to  last  for  longer  than  three  weeks  when 
experiment  planners  must  squeeze  in  a  final  training  week  before  two  weeks  of 
experiment  play.  Even  though  planners  extended  MN  LOE  II  through 
workshops  and  ad  hoc  events  to  conduct  portions  of  the  experiment  as  well  as 
initial  player  training,  MN  LOE  II  still  ran  out  of  time.  No  time  translated  into 
an  inability  to  explore  prime  experiment  questions,  such  as  the  discovery  of 
multinational  issues  associated  with  different  national  perspectives.  Here, 
players  had  insufficient  time  to  debate  national  viewpoints  regarding  ONA 
viability  during  ONA  development.  Instead,  time  was  used  during  the 
experiment  to  develop  ONA  process  procedures,  which  could  have  been 
accomplished  before  the  experiment  and  refined  during  the  training  week  in  a 
much-needed  rehearsal  walkthrough.  But  the  training  week  was  spent 
repeating  basic  training  on  tools  and  concepts  for  late-arriving  players.  The 
experiment  was  designed  to  examine  the  ONA  process,  not  to  generate 
products.  This  was  stated  clearly  throughout  the  planning  process.  However, 
during  execution,  some  participants  were  more  interested  in  producing 
products  than  in  working  the  process.  This,  along  with  overly  ambitious 
experiment  objectives,  caused  some  significant  time-related  problems.  Time 
constraints  seen  were  directly  correlated  to  the  factor  that  In  addition,  the 
player  list  was  unstable  right  up  to  the  beginning  of  the  experiment,  causing 
additional  repetitive  training  and  subsequent  time  constraints. 

□  Lessons  Learned:  Limit  objectives  by  the  amount  of  time  available  to  examine 
them.  Make  time  a  more  significant  factor  in  experiment  planning.  Also, 
identify  the  experiment  audience  prior  to  the  experiment,  and  ensure  that  they 
are  available  for  all  training. 

□  Recommendation:  Time  management  issues  must  be  considered  during 
experiment  planning  to  ensure  that  experiment  objectives  and  issues  are 
addressed.  Develop  a  more  definitive  timeline  of  experiment  events  tied  to 
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experiment  objectives.  A  mechanism  is  needed  to  make  time  available  for  issue 
exploration  that  does  not  constrain  progress  toward  insights,  but  that  also 
tenninates  exploration  when  acceptable  conclusions  are  reached.  Identify  the 
experiment  audience  as  early  as  possible,  and  ensure  that  they  attend  all  training. 

5.  Title:  Training  without  Practice  Walkthrough 

□  Observation:  Because  no  training  vignette  occurred,  players  perceived  that 
training  was  conducted  piecemeal,  without  revealing  how  all  the  pieces  fit 
together  until  late  in  the  game. 

□  Discussion:  After  the  experiment,  players  complained  that  they  “needed  a 
training  vignette  during  training  week  that  pulled  everything  together  so  they 
could  learn  how  to  perfonn  unified  tasks  instead  of  disconnected  steps.”  A 
training  vignette  was,  in  fact,  considered  and  planned,  but  was  discarded  when 
the  nations  unexpectedly  requested  additional  training/briefs.  As  a  result,  the 
first  vignette  was  essentially  on-the-job  training,  which  precluded  more  issue 
exploration.  In  hindsight,  a  practice  vignette  would  have  revealed  some 
previously  invisible  experimentation  bugs. 

□  Most  pre-experiment  training  for  players  focused  on  using  the  collaborative 
tool,  the  database,  various  other  tools,  the  ONA  concept,  and  the  experiment 
scenario.  This  training  was  spread  out  over  several  months  and  then  repeated 
during  training  week  for  the  benefit  of  late  arrivals.  ONA  process  procedures 
were  developed  and  training  occurred  during  the  experiment.  Training  in 
information-sharing  procedures  never  occurred  at  all. 

□  Lessons  Learned:  A  practice  walkthrough  is  essential  to  make  training 
meaningful  and  is  inevitable  before  or  during  the  experiment.  A  walkthrough 
reveals  hidden  problems  prior  to  the  experiment. 

□  Recommendation:  Plan  and  allocate  time  for  a  full  practice  walkthrough  to 
make  training  meaningful  and  to  debug  experiment  methodology. 

6.  Title:  Collaboration  Concept  of  Operations  (CONOPS)  or  Tactics,  Techniques, 
and  Procedures  (TTP) 

□  Observation:  The  process  of  building  an  ONA  in  a  multinational  environment 
suffered  without  a  concept  of  operations  for  collaboration. 

□  Discussion:  Due  to  the  absence  of  sound  collaboration  practices,  participants’ 
unstructured  use  of  text  chat,  audio,  discussion  threads,  file  naming 
conventions,  and  file  storage  practices  made  information  sharing  more  difficult 
than  it  should  have  been. 

□  Lesson  Learned:  Effective  collaboration  requires  well-thought-out  tool  use. 
CONOPS  and  TTPs  for  collaboration  are  needed. 

□  Recommendation:  Future  experiment  events  will  require  more  thought  and 
consideration  of  collaboration,  and  CONOPS  and  TTPs  should  be  developed 
for  future  collaboration  events. 
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7.  Title:  Collaborative  Experiment  Planning  Meetings 

□  Observation:  Coordination  meetings  often  lacked  multinational  participation  during 
the  experiment  planning  process  because  national  LNOs  had  many  obligations. 
Experiment  analysts  solved  this  problem  through  weekly  collaborative  sessions. 

□  Discussion:  Planning  meetings  often  did  not  include  multinational 

representation  because  national  LNOs  had  many  obligations.  Additionally,  key 
individuals,  groups,  or  their  representatives  were  often  absent.  The  experiment 
analysts  solved  this  problem  in  a  cost  effective  way  by  conducting  mandatory 
weekly  collaborative  sessions  via  Groove  and  teleconference.  All  nations  were 
intimately  involved  with  the  design  of  the  experiment  analysis  plan.  After  the 
experiment,  the  multinational  analysts,  many  of  whom  were  experiment  leads 
for  their  nations,  emphasized  the  importance  and  benefits  of  these  meetings 
and  strongly  recommended  that  the  same  be  done  for  follow-on  experiments. 

□  Lesson  Learned:  Collaborative  meetings  would  encourage  more  in-depth 
multinational  participation  and  would  result  in  a  more  robust  experiment. 
Partners’  direct  involvement  with  weekly  planning  meetings  would  reveal 
more  of  the  experiment  flaws  earlier  in  the  design  process.  The  continued 
involvement  of  partner  team  leads  during  experiment  planning  is  essential  for 
immediate  consideration  and  resolution  of  emerging  issues,  as  well  as  for  the 
prevention  of  oversights. 

□  Recommendation:  Design  experiment  planning  sessions  around  the  use  of  a 
collaborative  tool  or  a  teleconference  to  allow  all  multinational  experiment 
leads  to  be  more  directly  involved  with  the  planning  process.  Furthennore, 
require  mandatory  attendance  at  IPT  meetings  by  all  key  U.S.  participants  or 
their  representatives.  To  increase  non-U. S.  partners’  involvement  at  other 
planning  events,  hold  planning  meetings  at  non-U. S. -based  facilities,  thus 
seeking  greater  involvement  and  commitment  from  multinational  players. 

8.  Title:  CFBL  Net  Bandwidth  Needs  Boost  To  Support  Robust  MN  Experimentation 

□  Observation:  Despite  a  lack  of  modeling-and-simulation  support,  the  network 
barely  was  able  to  operate  with  existing  bandwidth  to  CFBL  net  points  of 
participation. 

□  Discussion:  Robust  experiments,  complete  with  modeling-and-simulation 
support,  can  use  up  to  10  Mbps  of  bandwidth,  based  on  past  experimentation 
results.  Existing  bandwidth  serving  CFBL  net  points  of  participation  is  only  1.5  to 
2.9  Mbps.  If  joint  experimentation  plans  to  experiment  in  a  robust  manner  with 
multinational  partners,  then  available  bandwidth  will  need  a  marked  boost. 

□  Lesson  Learned:  Robust  experimentation  with  multinational  partners  will 
require  a  boost  in  bandwidth  to  CFBL  net  points  of  participation. 

□  Recommendation:  Initiate  actions  to  boost  bandwidth  at  national  points  of 
participation. 


65 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


66 


VI.  CONCLUSION 


Effects-based  operations  will  define  the  future  of  joint  military  actions  that  depict 
advanced  warfighting  capabilities  across  the  full  spectrum  of  operations.9  Integrated  and 
holistic  information  synchronization  is  employed  to  view  the  adversary  from  a  system-of- 
systems  perspective.  Using  this  approach,  political,  military,  economic,  social, 
infrastructure,  and  information  (PMESII)  elements  are  linked  with  diplomatic, 
information,  military,  and  economic  (DIME)  actions  and  resources  to  improve  our 
understanding  of  the  adversary  and  to  disseminate  infonnation  to  commanders — all  to 
improve  decision  quality  and  to  increase  situational  awareness. 

Decision  superiority  is  a  critical  component  of  Joint  Vision  2020  that  aims  to 
achieve  full  spectrum  dominance  for  U.  S.  military  operations.10  Technological  advances 
that  enable  materiel  development  and  artificial  intelligence  systems  support  this 
dominance.  Yet  the  human  element  remains  the  nucleus  of  the  command-and-control 
system  and  must  be  supported  by  an  integrated  information  environment  that  is  capable  of 
fusing  diverse  and  dynamic  data  points. 

The  future  of  U.S.  military  operations  continues  to  involve  multinational 
coalitions,  whether  for  counter  terrorism  activities,  peacekeeping,  humanitarian  actions,  or 
warfighting  operations.  Using  this  coalition  mindset,  an  alliance  must  be  able  to  collect 
and  integrate  infonnation  elements  to  form  a  coherent  knowledge  capacity  for  decision¬ 
makers.  This  introduction  of  a  distributed  and  collaborative  information  fusion  and 
planning  effort  would  be  constrained  by  cultural  and  organizational  challenges  imposed  by 
the  coalition  perspectives  and  military  experiences. 

MN  LOE  II,  collaboratively  developed  by  the  participating  nations  and 
organizations,  tested  the  ability  of  a  multinational  coalition  to  build  and  fuse  infonnation 
elements  into  an  effects-based  database.  This  process  was  conducted  in  a  distributed 
collaborative  environment  using  the  operational  net  assessment  (ONA),  an  effects-based 
tool.  Experiment  participants  included  Australia,  Canada,  Gennany,  the  United  Kingdom, 
the  United  States,  NATO  CDE,  as  well  as  active  observers  from  France,  the  sixth  MIC 
member  nation  in  this  experiment.  With  the  exception  of  SACLANT,  countries  participated 
from  their  national  experiment  sites  using  collaborative  information  technology. 

The  ONA  enables  decision  superiority.  The  system-of-systems  analysis,  which 
focuses  on  PMESII  elements,  uses  collaboration  technologies  and  subject-matter  expertise 


9  U.  S.  Joint  Forces  Command.  (2002).  Toward  a  joint  warfighting  concent:  Rapid 

Decisive  Operations.  Suffolk:  J9  Joint  Futures  Lab. 

10  U.  S.  Joint  Forces  Command.  (2002).  Operational  Net  Assessment:  A  functional 

National  defense  construct.  Suffolk:  J9  Joint  Futures  Lab. 
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to  provide  decision-makers  with  a  rapid  and  holistic  view  of  the  adversary’s  strengths  and 
weaknesses,  as  well  as  coalition  DIME  resources,  to  affect  the  adversary.  System-of- 
systems  analysts  and  planners  collaborate  to  build  the  ONA  database  by  identifying  fused 
PMESII  effects,  nodes,  actions,  and  resources  that  provide  superior  knowledge  to  a 
decision-maker  who  projects  elements  of  national  power  to  achieve  specific  effects. 
Information  development  must  be  a  dynamic  process  to  keep  pace  with  changing 
conditions,  such  as  the  election  of  a  new  president  or  the  replacement  of  a  diplomatic 
minister.  The  effects-based  planning  element,  supported  by  the  ONA,  depends  on 
linkages  among  nodes,  actions,  and  resources  to  support  a  desired  effect.  This  experiment 
examined  coalition  members’  efforts  to  develop  an  ONA  and  to  create  linkages  to  support 
a  commander’s  intent  in  a  distributed  collaborative  setting. 

Based  on  favorable  results  from  MN  LOE  I  &  II,  the  future  looks  promising.  The 
development  of  a  coalition  operational  net  assessment  capability  is  highly  relevant,  as  is 
further  experimentation  on  information- sharing  procedures  in  the  multinational  arena.  All 
partners  involved  with  MN  LOE  II  considered  a  multinational  ONA  to  be  a  powerful 
process  that  includes  national  perspectives  and  that  uses  the  cultural  synergy  of 
collaboration.  This  experiment  clearly  demonstrated  the  stress  that  a  commercial 
technology  bears  to  support  distributed  collaboration  when  a  large  group  of  participants 
accesses  and  modifies  a  large  ONA  database.  However,  despite  these  technical 
difficulties,  distributed  collaboration  effectively  conducted  ONA  development. 

Additionally,  the  ability  to  share  information  among  coalition  partners  is  essential, 
even  in  a  pre-crisis  situation.  Multinational  information  sharing  is  key  to  building  trust 
and  confidence  in  a  coalition.  Furthermore,  the  investment  in  and  use  of  the  collaborative 
information  environment  must  continue.  The  capability  of  the  collaborative  system  used 
in  an  experiment  and  in  the  field  should  align  with  the  functional  requirements  of  the 
experiment  participants  or  of  the  combatant  command. 


68 


APPENDIX  A.  ANALYSIS  WORKSHOP  FEEDBACK 


The  experiment  analyst  workshop,  March  24-28,  2003,  in  an  extremely  aggressive 
schedule,  aimed  to  review  all  independent  analysis  completed  by  each  partner  nation 
analysis  team  and  to  prepare  the  MN  LOE  II  Final  Report.  Each  partner  nation  discussed 
lessons  learned,  as  well  as  independent  analysis  of  the  COIs  and  objectives.  Additionally, 
future  analysis  was  addressed.  The  lead  experiment  analyst  for  each  country  planned: 

□  To  present  and  discuss  analytical  results  of  MN  LOE  II 

□  To  offer  professional  insight  into  partner  nation  assessments 

□  To  show  draft  findings  and  lessons  learned  for  the  final  report 

□  To  work  towards  completing  the  final  report. 

The  workshop  was  a  success,  and  such  sessions  should  be  built  into  the 
experimentation  process.  The  face-to-face  meeting  produced  a  superior  exchange  of 
information  with  ample  discussions  in  a  balanced  atmosphere,  and  national  representations 
resulted  in  an  extremely  productive  working  group.  Many  multinational  experiment 
analysts  independently  examined  the  experiment  and  came  up  with  corroborating  results, 
lending  credence  to  the  MN  LOE  II  Final  Report.  Furthermore,  the  multinational 
collaboration  made  the  final  report  more  than  a  U.S. -centric  document. 

Many  lessons  learned  were  addressed  as  a  direct  result  of  the  workshop.  See 
Section  V  and  Appendix  H  for  the  entire  compilation  of  lessons  learned.  In  all  future 
experiments,  experiment  analysts  must  work  in  conjunction  with  the  experimentation 
design  and  planning.  Also,  the  constant  repetitiveness  seen  in  MN  LOE  II  must  be 
avoided  because  it  detracts  from  the  main  effort  of  analyzing  the  experiment. 
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APPENDIX  B.  SCENARIO  AND  VIGNETTE  DETAILS 


The  following  annex  contains  infonnation  relating  to  the  scenario  ramp-up  document 
and  the  four  vignettes  used  during  Multinational  Limited  Objective  Experiment  II.  The  ramp- 
up  document  was  posted  at  the  beginning  of  Week  1  in  the  Groove  shared  space  for  the 
coalition.  At  the  start  of  each  vignette’s  two-day  experiment  period,  the  vignette  briefing  and 
associated  Word  document  also  were  posted  in  the  coalition  shared  space. 
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A.  ANNEX  1  TO  APPENDIX  B  -  SCENARIO  RAMP-UP  DOCUMENT 


(U)  The  World  View  in  2010 

(U)  In  2010,  the  world  continues  in  an  uneasy  state  of  near-equilibrium,  with  indications 
of  economic,  political,  and  social  stresses.  The  armed  forces  of  the  U.S.  and  its 
multinational  allies  continue  to  be  stretched  to  react  to  potential  flashpoints,  as  well  as  to 
the  continuing  global  war  on  terrorism. 

(U)  The  emerging  global  economic  environment  created  new  vulnerabilities  for  the 
national  security  of  nations.  Energy  and  resources  continue  to  have  major  strategic 
influence  as  demand  increases  at  a  pace  that  exceeds  availability,  and  the  world’s  energy 
remains  petroleum  based.  New  technologies  divide  the  world  as  well  as  unite  it, 
simultaneously  creating  new  vulnerabilities  as  they  offer  opportunities.  All  physical  and 
virtual  borders  are  more  porous,  some  are  bending,  and  some  are  breaking.  Space  is  a 
critical  and  competitive  military  environment. 

(U)  The  U.S.  remains  the  premier  global  superpower,  and  with  the  European  Union, 
Russia,  China,  and  the  U.N.,  they  seek  to  maintain  a  peaceful  world  order.  Although  the 
U.S.  has  no  peer-competitor  with  regard  to  the  capabilities  and  means  to  execute 
sustained,  global  military  operations,  the  European  Union,  China,  and  Russia  can  conduct 
short-duration  military  operations  globally,  unilaterally,  or  with  only  minimal  assistance 
from  allies.  Within  the  past  several  years,  both  China  and  the  European  Union  have 
expanded  their  capabilities  to  complement  the  huge  technological  lead  of  the  U.S.  military. 

(U)  On  a  global  scale,  unemployment  is  gradually  improving,  but  a  great  disparity  exists 
between  the  “haves”  and  “have-nots.”  Many  nations  view  free  trade  and  a  global  financial 
system  as  the  optimal  solution  for  improving  their  economies.  Three  major  trading/economic 
blocs  control  the  global  economy — the  U.S.,  the  European  Union,  and  Asia,  largely  China  and 
Japan.  In  addition,  India  is  emerging  as  a  major  global  economic  power. 

(U)  The  emerging  global  economy  includes  much  economic  interaction,  and  markets  are 
generally  open.  However,  trade  barriers,  quotas,  export  restrictions,  and  occasionally 
sanctions  remain,  as  many  nations  seek  to  guide  their  own  economic  recovery  and  to  protect 
their  own  economies.  Many  nations  and  corporations  engage  in  economic  warfare,  which  is 
beginning  to  affect  certain  sectors  of  the  recovering  global  economies.  Global  economic 
demands  continue  to  push  the  limits  of  state-of-the-art  infonnation  and  technology  systems,  as 
competing  businesses  and  organizations  seek  to  gain  and  maintain  the  competitive  edge. 

(U)  Tensions  in  2010 

(U)  The  world  in  2010  holds  many  potential  flashpoints:  the  Middle  East,  Kashmir, 
possible  revolution  in  Venezuela,  ethnic  strife  and  terrorism  that  threaten  the  West  African 
states,  continuing  terrorism  and  insurgency  in  the  Caucasus,  and  the  ever-present  concerns 
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in  the  Persian  Gulf.  As  the  world  economies  continue  to  accentuate  the  differences  in 
classes  over  the  past  decade,  terrorism  took  root  in  more  countries  than  ever  before,  and 
terrorists  adopted  high-technology  tools  to  wage  their  asymmetric  war. 

(U)  These  sections  summarize  the  significant  events  that  have  shaped  the  world,  beginning  with 
the  supranational  organizations  and  continuing  in  more  detail  to  selected  individual  countries. 

(U)  The  Supranational  Organizations 

(U)  The  U.N.  gained  greater  relevance  through  its  ongoing  humanitarian  relief  and  peace 
operations,  gaining  political  leverage  with  numerous  affected  nations.  Additionally,  the 
U.N.  had  to  come  to  grips  with  the  increasing  worldwide  problem  of  internally  displaced 
persons  and  refugees.  The  U.N.  High  Commissioner  on  Refugees  (UNHCR)  and  the 
International  Organization  for  Migration  (IOM)  exerted  considerable  influence  in  several 
cases  of  ethnic  warfare.  However,  the  U.N.  is  still  affected  by  some  degree  of  distrust,  as 
well  as  the  occasional  “veto”  wars  in  the  U.N.  Security  Council. 

(U)  The  European  Union  is  evolving  into  an  increasingly  powerful  body  for  Europe,  ready  to 
progress  beyond  a  strictly  economic  entity  with  the  advent  of  more  political  unanimity  and  a 
notional  military  self-defense  pact.  However,  member-states  still  exercised  sovereignty  when 
it  was  in  their  best  interests.  The  European  Union  attempted  to  challenge  the  U.S. 
economically  as  a  peer  competitor,  but  it  remains  hobbled  by  high  labor  and  social  costs. 

(U)  Regions  and  states 

(U)  Middle  East.  The  Middle  East  remains  unstable.  Terrorism,  especially  by  Muslim 
extremists,  is  widespread  and  is  a  major  destabilizing  factor.  The  region  remains  militarized, 
and  military  expenditures  remain  high.  Intelligence  analysts  believe  that  more  than  one  nation 
has  nuclear  weapons  and  the  ballistic  and  cruise  missiles  to  employ  them.  The  U.S.  continues 
to  maintain  a  close  political,  economic,  and  military  relationship  with  Israel. 

(U)  Caucasus.  The  insurgencies  of  the  early  2000s  have  continued;  however,  in  2008, 
Russia  reacted  with  a  major  show  of  force  in  the  Caucasus  and  has  had  to  essentially 
maintain  an  army  of  occupation  ever  since.  The  insurgents  have  created  major  problems 
in  the  region  and  have  threatened  the  Trans-Caucasus  pipelines.  This  has  become  a  major 
issue  for  Russia  and  is  occupying  much  of  their  leadership’s  attention.  This  has  a 
spillover  effect  on  the  Russian  oilfields  in  central  Asia. 

(U)  Southwest  Asia 

(U)  The  U.S.  is  the  dominant  stabilizing  external  force  in  Southwest  Asia  and  maintains  a 
significant  military  presence  there.  The  U.S.  continues  to  provide  political,  economic, 
military,  humanitarian,  and  other  nation-building  support  to  Iraq  after  successfully 
affecting  the  change  of  regime.  Russia  has  exerted  influence  from  the  north,  and  China 
and  Japan  have  both  increased  their  presence  there. 
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(U)  Militant  fundamentalists  and  terrorist  organizations  remain  an  active  force  in  the 
region.  Yemen  has  resurfaced  as  a  center  of  terrorist  activity. 


(U)  Africa 

(U)  Several  states  in  Africa  have  virtually  ceased  to  exist  as  viable  nations — Rwanda, 
Somalia,  Liberia,  the  former  Ivory  Coast — and  their  populations  face  mass  starvation. 
The  UNHCR  and  multiple  nongovernmental  organization  relief  agencies  have  been 
relatively  ineffective  dealing  with  this  problem. 

(U)  Major  offshore  oil  fields  off  of  West  Africa  have  been  threatened,  alternately,  by 
competing  revolutionary  groups  or  the  military.  Several  acts  of  sabotage  or  terrorism  have 
resulted  in  damage  to  a  major  offshore  tanker  loading  port  and  have  driven  up  the  price  of 
African  crude  oil. 

(U)  A  vacuum  exists  in  Africa  due  to  the  centuries-long  deadly  effects  of  the  AIDS  virus 
and  other  diseases,  further  complicated  by  lack  of  medical  support  and  education  in  this 
region.  Some  nations  have  taken  advantage  of  this  situation  to  expand  their  presence  in 
Africa  in  order  to  acquire  and  control  natural  resources.  As  a  result,  Africa  has  become  a 
major  recruiting  base  for  terrorist  and  criminal  organizations. 

(U)  Americas.  The  majority  of  the  population  remains  in  the  barrios,  getting  poorer  and 
more  distant  from  the  opportunities  of  the  early  21st  century.  Additionally,  many  countries 
in  the  region  still  lack  the  resources  necessary  to  move  beyond  a  commodity-based 
economy.  The  major  security  threats  to  the  regional  states  have  generally  not  been  from 
their  neighbors,  but  rather  from  domestic  insurgencies,  drug  trafficking,  organized  crime, 
and  natural  environmental  disasters,  such  as  hurricanes  and  earthquakes. 

(U)  Asia 

(U)  North  and  South  Korea  united  in  a  single  peninsular  state  in  2006  after  North  Korea 
imploded,  largely  as  a  result  of  ongoing  famines  and  economic  chaos.  The  new  nation 
adopted  the  South  Korean  form  of  government  and  economic  system.  Concern  of  a 
negative  impact  on  the  economy  of  the  south,  as  they  combined  with  the  north,  was  well 
founded.  Without  a  Korean  “Marshall  Plan”  instituted  by  the  U.S.,  recovery  would  have 
taken  much  longer.  Korea  retains  close  security  ties  to  the  U.S.  to  ensure  its  national 
security,  and  a  greatly  reduced  U.S.  military  presence  remains  in  country.  Korea 
strengthened  its  political  and  economic  ties  with  Japan  and  Taiwan. 


(U)  China.  In  recent  years,  the  government  of  China  has  experienced  challenges  and 
disruptions  to  its  plan  of  continued  growth  and  has  widened  its  sphere  of  influence.  The 
transition  to  capitalism  has  created  huge  stresses  within  the  economy  and  society  and  has 
threatened  the  political  order.  An  internal  conflict  exists  between  the  disenfranchised 
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interior  sections  and  the  more  industrial  coastal  regions,  causing  China  to  become  much 
more  internally  focused. 

(U)  Despite  this,  China  possesses  the  strongest  and  largest  economy  and  military  in  Asia. 
China  maintains  strong  commercial  ties  with  much  of  the  world,  especially  with  the 
Pacific  Rim.  Though  lacking  the  sophistication  of  U.S.  technology,  China  has  become  a 
major  exporter  of  military  arms  worldwide. 

(U)  China’s  size,  its  nuclear  capability,  its  potential  to  develop  large  military  forces,  its 
uncompromising  stance  on  defending  its  ocean  frontiers,  its  domestic  political  uncertainties, 
combined  with  its  potential  for  internal  chaos,  make  it  a  factor  in  any  Asian  engagement. 

(U)  Subnational  Entities:  Terrorism,  Organized  Crime,  Pirates,  Black  Markets,  and 
Multinational  Corporations 

(U)  Terrorism  continues  to  be  a  major  destabilizing  factor  in  world  events.  Middle 
Eastern  terrorists  continue  to  lash  out  against  non-Islamic  influences  in  the  Middle  East. 
Saudi  Arabia,  Qatar,  and  Kuwait  have  become  less  stable  in  the  face  of  rising  “anti¬ 
infidel”  sentiment,  and  an  overthrow  of  any  of  these  governments  could  result  in  renewed 
large-scale  conflict  in  the  region. 

(U)  The  U.S. -led  war  on  terrorism  continues  across  much  of  the  world,  with  significant 
activity  in  the  recognized  hotspots.  Due  to  their  high  standard  of  living  and  open  societies, 
the  U.S.,  the  European  Union,  and  Japan  are  the  principal  targets  for  terrorist  attacks. 

(U)  Organized  crime  is  prospering.  The  continuing  demand  for  narcotics  maintains  a 
booming  market  for  the  drug  cartels.  Opiates  and  narcotics  are  sold  by  criminal  organizations 
that  emanate  from  South  America  and  Southern  Asia.  Drug  cartels  have  successfully 
established  processing  and  distribution  infrastructures  in  most  nations,  including  the  U.S. 

(U)  Both  terrorists  and  organized  crime  are  adept  at  exploiting  high  technology  in  their 
operations  and  possess  adroit  business  acumen  in  the  management  of  their  operations. 
Both  employ  advanced  and  sophisticated  weapons  and  have  become  formidable  foes. 

(U)  Piracy  has  been  increasing  worldwide,  reaching  new  levels  of  sophistication,  brazenness, 
and  brutality.  The  pirates  are  better  equipped  with  modem  weaponry  and  communications 
and  know  of  shipping  and  law  enforcement  activities.  Unsupported  suspicions  exist  of 
collusion  among  pirates,  governments,  and  law  enforcement  agencies  of  several  nations  that 
border  strategic  sea  lines  of  communication.  As  an  adjunct  to  piracy,  human  trafficking  has 
grown,  both  as  a  source  of  virtually  free  labor  and  as  a  solution  to  ethnic  disagreements. 

(U)  Discovered  Oil 

(U)  Multinational  corporations  enabled  rapid  economic  growth  with  their  quick 
development  of  the  newly  discovered  oil  and  gas  fields.  In  fact,  intense  competition 
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existed  among  corporations  to  secure  development  rights.  As  the  situation  evolved,  both 
nations  of  interest  saw  that  it  was  to  their  own  advantage  to  cooperate  to  gain  the  greatest 
possible  advantage  from  the  oil  corporations.  In  return,  the  corporations  agreed  to  large 
prepayments  and  huge  infrastructure  development  programs.  The  governments  of  both 
nations  of  interest  saw  basic  national  improvements  as  essential  first  steps,  and 
concurrently  began  planning  for  improvements  in  their  defense. 

(U)  Nationalist  economic  policies  continued  to  be  promoted.  In  addition  to  the  existing 
banking  and  financial  structure  run  by  a  small  Chinese  middle  class,  the  government 
promoted  indigenous  small  and  medium  businesses,  and  encouraged  the  establishment  of 
Islamic  Banks  and  the  continued  faith  in  an  antiquated  network  of  cooperatives. 

(U)  Political  Change 

(U)  The  military  remains  an  essential  factor  to  gain  and  maintain  political  power  in  both 
nations  of  interest  They  implemented  reforms  to  become  more  professional  and  less 
political.  As  a  result  of  the  overall  increases  in  prosperity  and  budgets,  as  well  as 
competition  from  foreign  and  domestic  corporations,  the  military  reduced  their 
commercial  operations. 

(U)  Military  Changes 

(U)  Early  in  2003,  the  national  leaderships  of  both  countries  of  interest  convinced  their 
respective  militaries  that  it  was  in  their  own  and  the  national  best  interests  to  transition 
from  a  public  security  police-like  force  to  a  true  national  defense  force.  This  was  due  to 
the  increased  need  to  protect  the  offshore  resources  from  foreign  encroachment.  To  that 
end,  the  military  had  to  project  a  more  professional  image.  In  concert  with  this,  and  as 
incentive  to  the  military,  the  government  agreed  to  acquire  force  structure  and  equipment 
to  match  the  new  security  role,  including  to  high-performance  coastal  defense  cruise 
missiles,  patrol  boats,  and  diesel  submarines.  In  exchange  for  this,  the  military  agreed  to 
begin  divesting  themselves  of  commercial  interests. 
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APPENDIX  C.  EXPERIMENT  CONTROL  DESCRIPTION 


A.  PERSONNEL 

MN  LOE  II  control  personnel  consisted  of  a  core  cadre,  backed  by  trusted  agents  in 
each  of  the  participating  nations.  This  core  group  included: 

□  U.S.  experiment  director 

□  U.S.  chief  controller 

□  National  controllers  in  each  country/NATO  SACLANT  (a  total  of  five) 

□  U.S.  planner  lead 

□  U.S.  SoSA  lead 

□  Functional  leads. 

Additional  personnel  provided  “white  cell”  support  in  monitoring  the  different 
collaborative  shared  spaces,  along  with  backing  up  primary  control  personnel.  Each 
control  position  was  identified  in  Groove  with  a  “CC”  prefix  prior  to  the  station  number. 

B.  EXECUTION  CONTROL  VIRTUAL  HEADQUARTERS 

A  separate  space  was  set  up  within  the  Groove  collaborative  architecture  to 
provide  the  control  cell  with  a  private  location  for  discussions  and  direction.  This  “CC” 
space  was  accessible  only  to  the  core  cadre. 

C.  EXPERIMENT  CONTROL  DAILY  BATTLE  RHYTHM 

The  four  experiment  days  per  week  were  eight  hours  in  length;  six  hours  were 
dedicated  to  actual  experiment  play,  and  the  remaining  time  to  LOE  preparation,  surveys, 
national  hot  washes,  and  experiment  control  hot  washes.  During  Week  1  on  Thursday, 
and  during  Week  2  on  Friday,  an  after-action  review  focused  on  the  senior  concept 
developers.  On  the  Fridays  of  Week  1  and  Week  2,  national  hot  washes  focused  on  the 
experiment  audience. 
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D.  U.S.  LNO  RESPONSIBILITIES 

A  representative  from  USJFCOM  J9  was  deployed  to  each  of  the  four  participating 
countries  to  help  facilitate  the  experiment.  In  addition  to  their  understanding  of  the 
experiment’s  mechanics,  these  personnel  received  extra  training  and  in-depth  briefings  on 
the  various  concepts  being  explored.  Representing  U.S.  interests,  and  not  part  of  the 
“formal”  host  nation  experiment  control  cadre,  the  LNOs: 

□  Worked  LNO-to-LNO  back-channel  communications  to  ensure  success  of 
national  lead  controller  success 

□  Provided  on-the-spot  U.S.  technical  assistance  to  their  national  team 

□  Provided  U.S.  assistance  to  the  national  lead  controller 

□  Provided  the  national  lead  controller  with  vignette  products  from  the  MN  LOE 
II  CD  for  action 

□  Acted  as  a  cultural  bridge. 

E.  NATIONAL  LEAD  CONTROLLER  DUTIES 

Each  country’s  national  lead  controller: 

□  Posted  national  “white  cell”  vignette  products  prior  to  the  start  of  the  vignette  (All 
vignette  products  were  posted  by  the  national  “white  cell”  LNO  at  the  end  of  the 
previous  experiment  day,  due  to  Groove/CFBL  net  bandwidth  limitations.) 

□  Determined  national  tool  and  concept  training  requirements 

□  Monitored  national  tool  and  concept  training  for  their  national  experiment 
audience  and  national  experiment  control  team 
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□  Monitored  daily  national  hot  wash 

□  Presented  results  of  daily  national  hot  wash  at  the  daily  experiment  control  hot  wash 

□  Facilitated  the  national  portion  of  the  after-action  review  during  Week  1  and  Week  2 

□  Supported  multinational  information-sharing,  concept-of-operations  play 

□  Ensured  that  national  systems,  spaces,  and  teams  were  ready  to  support  daily 
experiment  play 

□  Ensured  experiment  execution  in  accordance  with  the  experiment  control  plan 

□  Coordinated  the  national  part  of  in-focus/azimuth-check  sessions 

□  Advocated  survey/human  factors  participation. 

F.  MULTINATIONAL  LNO/NATO  SACLANT  RESPONSIBILITIES 

Each  participating  nation  and  NATO  SACLANT  sent  a  representative  to  the 
USJFCOM  J9  facility  in  Suffolk,  Va.,  who: 

□  Represented  national/NATO  SACLANT  interests 

□  Was  not  a  formal  part  of  experiment  control 

□  Provided  remote/local  assistance  to  the  national  teams 

□  Provided  assistance  to  the  U.S.  chief  controller 

□  Acted  as  a  cultural  bridge. 

G.  ROLE  PLAYING  AND  REQUESTS  FOR  INFORMATION 

National  white  cell  personnel  in  each  country  represented  national  entities.  For  the 
U.S.,  State  Department  and  the  Defense  Intelligence  Agency  were  represented.  Additionally, 
the  fonnal  request-for-infonnation  process  was  not  used  because  of  MN  LOE  II  manpower 
limitations.  White  cell  SoSAs  or  scenario  writers  answered  infonnation  queries. 

H.  VIGNETTE  EXECUTION 

The  process  undertaken  during  the  first  few  hours  of  each  vignette  included  these  steps: 

i.  Vignette  scenario  brief  was  given  by  the  U.S.  chief  controller  on  Groove  and 
posted  in  the  coalition  shared  space. 

ii.  Vignette  scenario  Word  document  was  posted  on  Groove  in  the  coalition  shared 

space. 

iii.  Supporting  products  were  extracted  by  the  J9  LNO  from  MN  LOE  II  CD  and 
were  provided  to  the  national  lead  controllers. 

iv.  The  national  lead  controller  notified  his  national  plans  lead  and  SoSA  lead  that 
new  products  have  been  posted  in  his  national  private  shared  space. 
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Week  1  “as  is”  MNIS: 


i.  The  products  were  posted  to  proper  MNIS  domain  spaces  as  identified  by  the 
security  classification  caveat,  i.e.,  ML,  TL,  BL1,  BL2,  coalition,  or  private. 

ii.  Further  product  distribution  required  national  disclosure  process  approval. 

iii.  If  approved  for  further  dissemination,  the  product  was  remarked/sanitized  and 
reposted  to  the  appropriate  MNIS  domain  space. 


Week  2  “future”  MNIS: 

i.  The  product  was  placed  in  the  proper  MNIS  domain  space  (coalition  or  private). 

ii.  Further  product  distribution  required  Coalition  Information-Sharing  Policy  and 
Procedures  Panel  disclosure  process  approval. 

iii.  The  national  plans  lead  and/or  SoSA  lead  determined  how  to  notify  their 
counterparts  in  the  various  MNIS  domains  of  the  new  national  information  that  is  being 
put  into  play  by  their  nation  (“as  is”  and  “future”  MNIS). 

I.  VIGNETTE  BATTLE  RHYTHM 

Four  vignettes  were  presented  to  the  experiment  audience  during  the  two-week 
experiment  execution  period.  Each  vignette  was  allocated  two  days  for  the  plans  and 
SoSA  personnel  to  review  and  act  on  the  injects/products  available  through  the  MNIS 
collaborative  process.  During  those  two  eight-hour  workdays,  six  hours  per  day  were 
spent  on  actual  product  review  and  ENAR  development. 
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J.  DAILY  EXPERIMENT  TIMES 


Because  of  the  distributed  location  of  the  participants,  the  eight-hour  experiment 
window  was  at  different  local  times  in  each  country  for  each  week.  This  allowed  all 
participants  to  experience  at  least  one  “normal”  duty  period  during  the  three-week  experiment. 
ZULU  times  were  used  during  the  event  for  all  control  and  scheduling  functions.  These  charts 
depict  how  the  eight-hour  window  cycled  through  the  different  countries  local  times: 
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K.  CLASSIFICATION  OF  EXPERIMENT  PARTICIPANTS  AND  OBSERVERS 

1.  Participant 

Participant  is  a  generic  term  for  LOE  member  nations  and  representatives  who 
collaborated  and  who  were  directly  involved  in  the  development  of  a  concept. 
Specifically,  participants: 

□  Are  represented  by  a  voting  member  within  the  LOE.  LOE  representation  is 
defined  as  “a  dictate  and/or  statement  within  a  formal  memorandum  of 
agreement,  memorandum  of  understanding,  or  official  assignment  to 
USJFCOM,  which  authorizes  participation  in  the  U.S. -sponsored  LOE  event.” 

□  Are  responsible  for  appropriate  LOE  national  planning,  programming, 
budgeting,  personnel  assignment,  operations,  administration,  analysis  and 
assessment,  or  any  combination  thereof 

□  Are  authorized  users  and/or  administrators  of  the  hardware,  software,  and 
networks  associated  with  the  LOE 

□  Have  full  access  to  classified  information,  based  on  right  and  need  to  know. 

2.  Observer 

Observer  is  a  generic  term  for  member  nations,  nonmember  nations,  or  alliances 
and  representatives  who  are  non-collaborators  in  the  development  of  a  concept.  Observers 
are  allowed  access  to  information  and  are  allowed  to  watch  the  execution  of  preapproved 
portions  of  the  LOE  process.  Specifically,  observers: 
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□  Are  member  nations,  nonmember  nations,  or  alliances  and  representatives  who 
have  been  preauthorized  to  possess  observer  status  by  all  voting  members  of 
the  LOE 

□  Watch  from  a  geographic  location  that  is  preapproved  by  all  voting  members 
of  the  LOE 

□  Participate  in  the  LOE  execution  phase  without  interfering.  For  example,  an 
observer  is  not  authorized  to  use  LOE  hardware,  software,  or  network,  and  is 
not  authorized  to  engage  in  formal  discourse  with  the  LOE  participants  during 
the  LOE  process. 

□  Are  not  authorized  to  use  radio  frequency  equipment,  such  as  cell  phone; 
magnetic  media,  such  as  recording  devices  and  computers;  electronic  devices, 
such  as  PDAs,  pens,  pencils,  and  paper;  or  any  other  item  not  preapproved  by 
all  voting  members  of  the  LOE 

□  Must  be  approved  by  all  voting  members  of  the  LOE  for  fonnal  receipt  of  any 
LOE  documentation  and  electronic  media 

□  Are  restricted  to  unclassified  information  access  only. 

3.  Visitor 

Visitor  is  a  generic  tenn  for  LOE  members,  non-member  nations,  or  alliances  and 
representatives  who  are  non-collaborators  in  the  development  of  a  concept.  Visitors  are 
allowed  limited  and  controlled  access  to  information  or  events  related  to  a  concept  or  LOE 
process.  Visit  requests  require  pre-approval  by  all  voting  members  of  the  LOE.  Visitors 
from  LOE  voting  and  nonvoting  member  nations  are  very  important  persons,  high-ranking 
officials,  and/or  other  personnel  who  temporarily  visit  a  portion  of  the  LOE  process. 
Those  from  member  nations  are  accorded  all  the  applicable  privileges  of  an  LOE 
participant  during  the  visit;  those  from  nonvoting  member  nations  must  comply  with 
observer  rules  and  restrictions  during  the  visit. 
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APPENDIX  D.  SENIOR  CONCEPT  DEVELOPER  PARTICIPATION 


A.  PURPOSE  OF  SCD  PROGRAM 

A  significant  element  of  warfighting  experimentation  is  the  participation  of  a  select 
group  of  former  general  and  flag  officers.  These  individuals  participate  in  a  variety  of 
activities  as  a  source  of  experience  and  knowledge  that  contributes  to  the  growing 
understanding  of  the  concepts  being  examined  during  the  experiment.  In  all  cases,  the 
intent  is  to  use  their  knowledge,  experience,  expertise,  and  high-level  influence  to  refine 
concepts  used  to  transform  the  military  forces  for  the  future. 

B.  SUMMARY  OF  SCD  PARTICIPATION 

Given  the  multinational  context  of  MN  LOE  II,  the  seven  senior  concept 
developers  who  participated  in  this  experiment  appropriately  were  from  each  of  the 
participating  nations — two  from  the  United  States,  two  from  Germany,  and  one  each  from 
Australia,  Canada,  and  the  United  Kingdom.  To  use  the  richness  of  their  individual  and 
collective  experiences  and  intellects  to  the  fullest,  they  served  as: 

□  Senior  advisors  to  the  experiment  director  during  Weeks  1  and  2 

□  Senior  observers  and  assessors  during  Weeks  1  and  2 

□  Participants  in  four  in-focus  sessions 

□  Participants  in  four  azimuth-check  sessions 

□  Participants  in  two  after-action  reviews 

□  Authors  of  detailed  written  assignments. 

C.  KEY  EVENTS 

The  SCDs  initially  participated  in  two  days  of  training,  January  15-16,  including 
briefings  on  the  ONA  and  MNIS  concepts,  SCD  responsibilities,  experiment  objectives 
and  construct,  and  the  use  of  Groove  as  the  collaborative  tool  to  support  experiment  play. 
This  short  period  of  time  also  allowed  the  SCDs  to  greet  each  other  personally — perhaps 
the  only  time  they  would  meet  face-to-face  during  the  globally  distributed  experiment. 
Even  with  no  prior  working  relationships,  they  quickly  engaged  in  several  high-level 
discussions  and  made  some  major  observations. 

Besides  observing  the  collaborative  activity  of  the  experiment  players  within  the 
various  Groove  spaces,  the  SCDs  spent  a  significant  portion  of  the  first  two  weeks  in 
collaboration  among  themselves.  This  generated  a  large  volume  of  written  material  that 
has  led  to  a  clearer  understanding  of  many  multinational  concept  issues.  These  exchanges 
not  only  prepared  the  SCDs  for — and  consequently  enriched — the  in-focus  and  azimuth 
sessions,  but  they  also  laid  some  of  the  groundwork  for  the  two-day,  post-experiment 
seminar  conducted  on  March  17-18. 
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Two  teleconference  in-focus  sessions  occurred  each  week  to  generate  discussion 
about  the  two  concepts  that  were  examined:  multinational  infonnation  sharing  and 
operational  net  assessment.  The  topic  and  timing  of  each  question  were  carefully 
coordinated  with  the  critical  operational  issues  of  the  experiment  activity  at  that  time. 
These  facilitated  sessions  encouraged  “out-of-the-box”  discussion  that  extended  concept 
exploration  beyond  the  scope  of  the  experiment  objectives. 

The  SCDs  also  participated  in  azimuth-check  sessions.  Like  the  in-focus  sessions, 
these  twice-weekly  teleconferences  offered  facilitated  discussion.  These  forums  allowed 
the  analysts  to  share  their  views  of  the  effectiveness  of  the  experiment  to  support  the 
objectives.  The  sessions  also  allowed  senior  concept  developers  to  share  their  insights 
regarding  the  critical  operational  issues. 

Two  formally  facilitated  and  teleconferenced  After  Action  Reviews  were  conducted 
during  the  experiment,  one  near  the  end  of  Week  1  and  a  Final  AAR  on  the  last  day  of  Week 
2.  While  the  primary  participants  in  the  AARs  included  the  National  Planners  and  SoSA 
Analysts,  the  SCDs  again  were  queried  for  their  thoughts  regarding  their  observed  strengths 
and  weaknesses  of  the  concepts  as  they  were  applied  in  the  experiment  and  also  their 
perceptions  of  the  attributes  of  the  experiment’s  design  and  control  apparatus.  Additionally, 
all  participants  were  challenged  to  provide  comments  on  concept-related  questions  that 
focused  on  the  week’s  experiment  activity.  It  was  in  this  instance  again  that  the  SCDs 
contributed  valuable  insights,  valuable  in  the  sense  that  their  assessments  were  created 
through  the  lens  of  their  unique,  collective  experience  and  understanding. 

During  the  March  17-18  meetings  of  the  senior  concept  developers,  they  reviewed  the 
analysts’  preliminary  assessments  and  provided  qualitative  input  based  on  their  collective 
participation.  Admiral  E.P.  Giambastiani,  Commander,  U.S.  Joint  Forces  Command,  enjoined 
them  to  identify  specific,  high-level  recommendations  for  his  immediate  action. 

D.  HIGH-LEVEL  SCD  RECOMMENDATIONS 

Senior  concept  developers  identified  two  levels  of  recommendations  for  action  by 
high-level  military  and  government  officials.  Level  1  recommendations  were  deemed 
essential  to  the  development  of  the  concepts;  they  require  the  personal  attention  and 
support  of  commander  of  USJFCOM.  Level  2  recommendations  should  be  brought  to  the 
commander’s  attention,  but  should  be  implemented  at  a  subordinate  level. 

Level  1  Recommendations 

1.  A  real-world  scenario  should  be  used  for  the  next  multinational  experiment, 
scheduled  for  February  2004.  All  nations  involved  in  the  experiment  could  provide  a 
working  database  to  address  the  associated  elements  of  DIME  and  PMESII.  Additionally, 
this  dynamic  scenario  would  be  beneficial  to  each  nation’s  knowledge  base  and  national 
activity  in  the  area.  The  senior  concept  developers  unanimously  and  emphatically 
endorsed  the  importance  and  value  of  this  recommendation.  Other  advantages  include: 
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□  Greater  detail  in  the  ONA  database 

□  Greater  stress  placed  on  the  ONA  process  and  on  the  MNIS  hypothesis 

□  Increased  political  support  and  multinational  interagency  participation 

□  Development  of  real-world  nodes,  links,  actions,  and  resources 

□  Generation  of  an  effects  plan  that’s  useful  to  experimentation  and  ongoing 
stability  operations. 

2.  The  commander,  USJFCOM,  should  advise  the  Department  of  Defense  to 
review  Directive  5230.11,  an  information-sharing  policy,  to  reflect  the  demanding 
requirements  of  the  information  age.  While  every  nation  would  have  a  foreign  disclosure 
policy  regarding  highly  classified  information,  the  philosophy  behind  this  directive  should 
emphasize  “withholding  infonnation  by  exception”  in  a  coalition  environment  for  joint 
doctrine  development. 

3.  A  knowledge  white  paper  should  be  developed  to  generate  further  study, 
debate,  and  exploration  of  this  critical  aspect  of  future  joint  and  combined  operations.  The 
paper  should  define  knowledge  and  should  address  the  supporting  elements  of  knowledge 
superiority,  management,  readiness,  and  warfare.  Although  current  joint  force  concepts 
emphasize  the  value  of  and  necessity  for  knowledge,  a  distinct  “knowledge  concept”  does 
not  exist  that  explores  knowledge-related  issues. 


Level  2  Recommendations 


1 .  The  concept  of  national  knowledge  advantage  centers  should  be  expanded.  The 
ONA  process  requirements  of  the  standing  joint  force  headquarters  should  be  used  as  a 
forcing  function  for  this  purpose. 

2.  The  multinational  collaborative  information  environment  should  begin  120  days 
prior  to  the  start  of  the  next  multinational  experiment,  allowing  a  realistic  development  of 
the  ONA  database.  In  addition,  it  would  facilitate  simultaneously  an  in-depth  exploration 
of  the  strengths  and  weaknesses  of  persistent  collaboration  over  time  and  space. 

3.  The  ONA  naming  convention  should  be  reexamined.  Although  the  ONA 
concept  was  considered  to  be  a  necessity,  its  nomenclature  is  confusing  to  our 
multinational  partners,  as  well  as  to  some  of  our  own  military  and  government  personnel. 
The  concept  development  pathway  should  embrace  both  the  principles  and  processes  of 
ONA;  however,  the  ONA  name  should  be  changed  to  “effects  analysis”  or  some  other 
commonly  understood  and  agreed-upon  term. 

4.  The  lessons  learned  from  this  experiment  on  multinational  information  sharing 
should  be  added  to  the  Pinnacle  Impact  03  experiment. 

5.  Efforts  should  continue  to  embed  a  simpler  and  more  effective  MNIS  concept 
into  joint  doctrine  development.  The  current  Draft  JP2-02  is  on  the  right  track. 
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APPENDIX  E.  SoSA  ACTIVITIES  LEADING  TO  MN  LOE  II 


•  Mid-August  2002  until  Workshop  I 

□  Conducted  background  research  on  and  system-of-systems  analysis  of  the 
countries  of  interest 

□  Developed  2002  PMESII  summaries  of  countries  of  interest 

□  Prepared  briefings  on  the  PMESII  summaries 

□  Participated  in  the  technical  development  of  the  new  database  for  MN  LOE  II; 
provided  thoughts  and  requirements  for  the  new  database 

□  Conducted  exploratory  meetings  for  ClearForest  use 

□  Participated  in  MN  LOE2  initial  planning  conference 

□  Developed  workshop  plan 

•  Workshop  I,  Sept.  30-Oct.  2,  2002 

□  Executed  Workshop  I  plan 

□  Developed  and  presented  ONA  system-of-systems  analysis  briefing 

□  Presented  PMESII  summary  briefs  for  both  countries  of  interest 

•  Oct.  3,  2002,  to  Workshop  II 

□  Continued  research  on  and  system-of-systems  analysis  of  countries  of  interest 

□  Supported  white  cell  scenario  and  LOE  construct  development 
o  Provided  background  material  for  scenario  refinement 

o  Supported  vignette  development;  reviewed/critiqued  drafts,  provided 
background  material 

o  Attended  weekly  white  cell  development  meetings 
o  Reviewed  proposed  LOE  construct 

□  Researched  and  wrote  2010  PMESII  summary  updates 

□  Initiated  node  selection  for  the  ONA  database 

□  Continued  to  support  technical  development  of  the  database 

□  Developed  Workshop  II  plan 

□  Developed  Workshop  II  ONA  training  briefing 

□  Prepared  PMESII  example  training  briefing  for  Workshop  II 

□  Prepared  2010  PMESII  update  briefings  for  Workshop  II 

□  Participated  in  MN  LOE  II  final  planning  conference 
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•  Workshop  II,  Nov.  5-7,  2002 

□  Executed  Workshop  II  plan 

□  Presented  ONA  training  briefing 

□  Presented  PMESII  example  training  briefing 

□  Presented  PMESII  update  briefings  for  both  COI 

□  Participated  in  development  of  red-blue,  blue -red,  red-red,  and  blue-blue  views 

□  Participated  in  ClearForest  familiarization  session 

•  Nov.  8  to  Workshop  III 

□  Continued  node  selection  for  the  ONA  database 

□  Provided  final  inputs  to  the  technical  development  of  the  database 

□  Researched  and  prepared  node  related  resources  for  the  database 

□  Began  entering  nodes  and  resources  into  the  database  (Dec.  2,  2002) 
o  Created  node  and  resource  identification  numbering  schemes 

o  Created  resource  documents  from  sources  (books,  articles,  Web  pages) 

□  Prepared  and  presented  briefings  on  ONA  process,  system-of-systems  analysis, 
database  entry,  PMESII  2002  summaries,  and  PMESII  2010  updates  for  NATO 
/SACLANT  participants 

□  Prepared  and  presented  briefings  on  ONA  process,  system-of-systems  analysis, 
database  entry,  PMESII  2002  summaries,  and  PMESII  2010  updates  for  UK 
participants 

□  Provided  database  user  expertise  to  support  multiple  ONA  database  training 
sessions 

□  Reviewed  white  cell  MSEL  injects  for  the  LOE 

□  Reviewed  final  white  cell  vignette  drafts 

□  Modified  and  entered  all  Australian,  Canadian,  and  German  products  into  the 
ONA  database 

o  Modified  the  nodes  to  match  U.S.  node  format  already  in  the  database 
o  Eliminated  redundancies  among  national  inputs 

o  Modified  Australian,  Canadian,  and  German  resource  inputs  so  database 
would  accept  the  resource  data 

o  Created  additional  resources  to  better  justify  Australian,  Canadian,  and 
German  nodes 

□  Commenced  node-to-node  linking  process 

□  Created  ONA  database  business  rules 
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o  Prepared  briefing  on  the  rules 


•  Workshop  III,  Jan.  13-17,  2003 

□  Executed  Workshop  III  plan 

□  Presented  briefings  on  SoSA  use  of  the  database  and  database  business  rules 

□  Participated  in  Groove  training 

□  Participated  in  wargaming  phase  of  the  baseline  ONA  development 

□  Participated  in  effects  development 

•  Workshop  III  to  MN  LOE  II 

□  Standardized  database  node  and  resource  identification  nomenclature 

o  Made  changes  to  Australian,  Canadian,  German,  and  UK  nodes/resources 
for  standardization  to  ease  database  search 

□  Continued  to  make  node-to-node  links 

□  Participated  in  ClearForest  training 

□  Corrected  Gennan  nodes  as  per  their  request 

□  Added  NATO  node/resource  data  into  database 
o  Resolved  conflicts  among  NATO  nodes 

o  Standardized  their  input  with  rest  of  the  database 
o  Created  resources  to  support  NATO  nodes 

□  Modified  ONA  SoSA  business  rules  as  per  Workshop  III  suggestions 

□  Prepared  ONA  database  SoSA  training  materials 
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APPENDIX  F.  DATA  COLLECTION  MATRIX 


The  data  collection  matrix  was  a  planning  tool  that  the  experiment  analysts  used  to 
organize  all  sources  of  data  in  one  location.  After  all  data  sources  were  listed,  analysts 
could  see  clearly  whether  each  experiment  objective  and  countries-of-interest  area  had 
been  addressed.  It  was  also  used  as  a  bookkeeping  tool  for  analysis.  The  matrix  contains 
measures  for  each  objective  and  COI,  as  well  as  respective  survey  questions.  It  also 
categorized  the  data  source  and  collection  frequencies.  Additional  indices  used  for 
bookkeeping  only  are  not  shown  in  this  appendix.  The  finalized  data  collection  matrix 
used  for  MN  LOE  II  is  listed  in  the  subsequent  pages. 
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Sequence 

No. 

OBS 

SVY 

C4I 

SCD 

Freq. 

1 

1 

Objective:  Identify  and  assess  issues  associated  with  the  ability  of  national 
headquarters  staffs  to  conduct  a  distributed  ONA 

2 

1.1 

COI:  Is  the  US  ONA  process  viable  in  a  coalition  environment? 

3 

1.1.1 

Measure:  Rating  of  viability  of  US  ONA  process  in  a  coalition  environment 

4 

Hypothesized  response:  The  US  ONA  process  is  viable  in  a  coalition  environment  despite 
differences  in  partner  organizations  and  command  relationships 

5 

1.1. 1.1 

Data  requirement:  Viability  of  US  ONA  process  in  a  coalition  environment 

6 

1.1. 1.1.1 

Data  element:  Survey  questions 

7 

1 .  The  US  ONA  process  is  viable  in  a  coalition  environment  (agree  scale) 

POC 

W1  W2 

8 

2.  The  ONA  process  improved  the  operational  planning  process  with  coalition  partners. 
(Improve  Scale) 

POC 

W2 

9 

3.  What  kinds  of  problems  did  players  spend  a  lot  of  time  trying  to  solve?  (Narrative) 

X 

PC 

W1  W2 

10 

1.1.2 

Measure:  Rating  of  impact  of  unequal  access  to  data  on  the  ONA  process 

11 

Hypothesized  response:  Unequal  access  had  an  impact  on  the  ONA  process  but  all 
participants  were  able  to  contribute  to  the  process. 

12 

1. 1.2.1 

Data  requirement:  ONA  users  perception  of  impact  of  unequal  access  to  ONA  data 

13 

1.1.2. 1.1 

Data  element:  Survey  questions 

14 

1 .  1  was  able  to  perceive  instances  where  it  was  apparent  that  participants  had  unequal  access 
to  data,  (true/false)  If  so  describe. 

X 

p 

W1  W2 

15 

2.  Unequal  access  to  ONA  data  had  a  significant  impact  on  the  ONA  process,  (agree  scale) 

POC 

W1  W2 

16 

3.  To  what  degree  do  you  feel  participants  could  provide  needed  information  to  other  members 
during  the  ONA  development  process?  (degree  scale) 

X 

W1  W2 

17 

1.1.3 

Measure:  Frequency  of  ONA  knowledge  base  access 

18 

Hypothesized  response:  Knowledge  base  access  may  change  as  participant's  understanding 
of  the  ONA  process  improves. 

19 

1. 1.3.1 

Data  requirement:  Number  of  knowledge  base  accesses  by  each  participant  in  the  coalition 
network  over  time 

20 

1.1.3. 1.1 

Data  element:  Number  of  knowledge  base  accesses  by  each  participant  in  the  coalition 

21 

1 .  Number  of  knowledge  base  accesses  by  each  participant  in  the  coalition  network 

X 

Hourly 

22 

1.1.4 

Measure:  Frequency  of  ONA  knowledge  base  update 

23 

Hypothesized  response:  Knowledge  base  updates  may  change  as  participant's  understanding 
of  the  ONA  process  improves. 

24 

1. 1.4.1 

Data  requirement:  Number  of  knowledge  base  updates  made  by  each  participant  in  the 
coalition  network  overtime 

25 

1.1.4. 1.1 

Data  element:  Number  of  knowledge  base  updates  made  by  each  participant  in  the  coalition 
network 

26 

1 .  Number  of  knowledge  base  updates  made  by  each  participant  in  the  coalition  network 

X 

Hourly 

27 

1.1.5 

Measure:  Frequency  of  collaborative  document  access 

28 

Hypothesized  response:  Collaborative  document  access  may  change  as  participant's 
understanding  of  the  ONA  process  improves. 

29 

1. 1.5.1 

Data  requirement:  Number  of  collaborative  document  accesses  by  each  participant  in  the 
coalition  network  over  time 
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Sequence 

No. 

OBS 

SVY 

C4I 

SCD 

Freq. 

30 

1.1.5. 1.1 

Data  element:  Number  of  collaborative  document  accesses  by  each  participant  in  the  coalition 
network 

31 

1 .  Number  of  collaborative  document  accesses  by  each  participant  in  the  coalition  network 

X 

Hourly 

32 

1.1.6 

Measure:  Frequency  of  collaborative  document  update 

33 

Hypothesized  response:  Collaborative  document  update  may  change  as  participant's 
understanding  of  the  ONA  process  improves. 

34 

1. 1.6.1 

Data  requirement:  Number  of  collaborative  document  updates  made  by  each  participant  in  the 
coalition  network  over  time 

35 

1.1.6. 1.1 

Data  element:  Number  of  collaborative  document  updates  made  by  each  participant  in  the 
coalition  network 

36 

1 .  Number  of  collaborative  document  updates  made  by  each  participant  in  the  coalition 

X 

Hourly 

37 

1.1.7 

Deleted 

38 

1.1.8 

Measure:  Rating  of  the  perceived  usefulness  of  ONA  process 

39 

1. 1.8.1 

Data  Requirement:  ONA  user's  perceptions  of  the  usefulness  of  ONA 

40 

1.1.8. 1.1 

Data  Element:  Survey  Questions  (all  7-point  Likert  Agree  Scale) 

41 

1 .  The  ONA  process  allows  me  to  accomplish  tasks  more  quickly. 

P 

Wed  Weekly 

42 

2.  Using  the  ONA  process  increases  my  productivity. 

P 

Wed  Weekly 

43 

3.  Using  the  ONA  process  improves  my  job  performance. 

P 

Wed  Weekly 

44 

4.  Using  the  ONA  process  enhances  my  effectiveness  on  the  job. 

P 

Wed  Weekly 

45 

5.  Using  the  ONA  process  makes  it  easier  to  do  my  job. 

P 

Wed  Weekly 

46 

6.  Overall,  1  find  the  ONA  process  useful  in  my  job. 

P 

Wed  Weekly 

47 

1.1.9 

Measure:  Rating  of  the  perceived  ease  of  use  of  ONA 

48 

1. 1.9.1 

Data  Requirement:  ONA  user's  perceptions  of  the  ease  of  use  of  ONA 

49 

1.1.9. 1.1 

Data  Element:  Survey  Questions  (all  7-point  Likert  Agree  Scale) 

50 

1 .  Learning  to  use  the  ONA  process  was  easy  for  me. 

P 

Wed  Weekly 

51 

2.  During  MNLOE  1  found  it  easy  to  get  the  ONA  process  to  do  what  1  wanted  it  to. 

P 

Wed  Weekly 

52 

3.  During  MNLOE  1  found  the  ONA  process  flexible  to  interact  with. 

P 

Wed  Weekly 

53 

4.  During  MNLOE  the  ONA  process  was  clear  and  understandable. 

P 

Wed  Weekly 

54 

5.  1  found  it  easy  to  become  skillful  at  using  the  ONA  process. 

P 

Wed  Weekly 

55 

6.  Overall,  1  find  the  ONA  process  easy  to  use. 

P 

Wed  Weekly 

56 

1.1.10 

Measure:  Rating  of  perceived  behavioral  control  of  ONA 

57 

1.1.10.1 

Data  Requirement:  ONA  user's  perception  of  their  behavioral  control  of  ONA 

58 

1.1.10.1.1 

Data  Element:  Survey  Questions  (all  7-point  Likert  Agree  Scale) 

59 

1 .  1  have  the  necessary  training  to  use  the  ONA  process. 

P 

Wed  Weekly 

60 

2.  The  training  provided  has  better  prepared  (assists)  me  to  use  the  ONA  process. 

P 

Wed  Weekly 

61 

3.  Help  from  colleagues  assists  me  to  use  the  ONA  process. 

P 

Wed  Weekly 

62 

4.  1  feel  1  have  the  necessary  skills  to  use  the  ONA  process. 

P 

Wed  Weekly 

63 

5.  1  feel  1  can  control  the  ONA  process  when  1  use  it. 

P 

Wed  Weekly 

64 

6.  Using  the  ONA  process  is  easy  for  me. 

P 

Wed  Weekly 

65 

1.1.11 

Measure:  Rating  of  ONA-related  work  processes 

66 

1.1.11.1 

Data  Requirement:  ONA  user's  perceptions  of  the  ONA  work  processes 

67 

1.1.11.1.1 

Data  Element:  Survey  Questions  (all  7-point  Likert  Agree  Scale) 

68 

1 .  1  believe  that  the  ONA  process  is  well  targeted  to  the  work  of  my  organization  as  a  whole. 

P 

Wed  Weekly 
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No. 

OBS 

SVY 

C4I 

SCD 

Freq. 

69 

2.  1  believe  that  the  ONA  process  is  well  targeted  to  the  work  of  my  team. 

P 

Wed  Weekly 

70 

3.  1  believe  that  the  ONA  process  is  well  targeted  to  the  organizational  needs  of  my 
organization  as  a  whole. 

P 

Wed  Weekly 

71 

4.  1  believe  that  the  ONA  process  is  well  targeted  to  the  organizational  needs  of  my  team  in 
MNLOE2. 

P 

Wed  Weekly 

72 

1.1.12 

Measure:  Rating  of  confidence  and  familiarity  with  ONA 

73 

1.1.12.1 

Data  Requirements:  ONA  user's  perceptions  of  their  confidence  and  familiarity  with  ONA 

74 

1.1.12.1.1 

Data  Element:  Survey  Questions  (all  7-point  Likert  Agree  Scale) 

75 

1 .  1  am  familiar  with  the  use  of  the  ONA  process. 

P 

Wed  Weekly 

76 

2.  1  am  confident  when  using  the  ONA  process. 

P 

Wed  Weekly 

77 

3.  1  believe  that  in  the  future  1  will  become  as  familiar  with  the  ONA  process  as  1  am  with  the 
existing  process. 

P 

Wed  Weekly 

78 

4.  1  believe  that  in  the  future  1  will  develop  as  much  confidence  in  the  ONA  process  as  1  have  in 
the  existing  process. 

P 

Wed  Weekly 

79 

1.1.13 

Measure:  Rating  of  perceived  information  supply  before,  during  and  after  the  ONA  process 
was  introduced 

80 

1.1.13.1 

Data  Requirements:  ONA  user's  perceptions  of  information  supply  before  ONA  was  introduced 

81 

1.1.13.1.1 

Data  Element:  Survey  Questions  (all  7-point  Likert  Agree  Scale) 

82 

1 .  The  information  supplied  to  me  to  date  about  the  ONA  process  kept  me  fully  informed. 

P 

Wed  Weekly 

83 

2.  The  information  supplied  to  me  to  date  about  the  ONA  process  was  suited  to  members  of 
my  team. 

P 

Wed  Weekly 

84 

3.  The  information  supplied  to  me  to  date  about  the  ONA  process  was  suited  to  my  particular 
role  in  my  organization. 

P 

Wed  Weekly 

85 

4.  Based  on  this  information  the  ONA  process  has  met  my  expectations. 

P 

Wed  Weekly 

86 

1.1.14 

Measure:  Rating  of  perceived  subjective  norms  in  relation  to  the  use  of  ONA 

87 

1.1.14.1 

Data  Requirements:  ONA  user's  perceptions  of  subjective  norms  in  relation  to  ONA 

88 

1.1.14.1.1 

Data  Element:  Survey  Questions  (all  7-point  Likert  Agree  Scale) 

89 

1 .  Generally,  people  in  my  organization  think  1  should  use  the  ONA  process. 

P 

Tue  W0  and  2 

90 

2.  Generally,  my  friends  at  work  think  1  should  use  the  ONA  process. 

P 

Tue  W0  and  2 

91 

3.  Generally,  my  superiors  think  1  should  use  the  ONA  process. 

P 

Tue  W0  and  2 

92 

1.1.15 

Measure:  Rating  of  ONA-related  help  from  colleagues 

93 

1.1.15.1 

Data  Requirements:  ONA  user's  perceptions  of  ONA-related  help  from  colleagues 

94 

1.1.15.1.1 

Data  Element:  Survey  Questions  (both  7-point  Likert  Agree  Scale) 

95 

1 .  1  have  found  it  easy  ask  my  colleagues  for  help  when  using  the  ONA  process. 

P 

Wed  Weekly 

96 

2.  My  colleagues  have  been  a  useful  source  of  information. 

P 

Wed  Weekly 

97 

1.1.16 

deleted 

98 

1.1.17 

Measure:  Rating  of  the  effect  of  ONA  on  work  practices 

99 

1.1.17.1 

Data  Requirements:  ONA  user's  perceptions  of  the  effect  of  ONA  on  work  practices 

100 

1.1.17.1.1 

Data  Element:  Survey  Questions  (all  7-point  Likert  Importance  Scale) 

101 

1 .  1  consider  that  the  work  of  my  organization  has  been  improved/worsened  by  the 
introduction  of  the  ONA  process. 

P 

Wed  Weekly 

102 

2.  1  consider  that  the  work  of  my  team  has  been  improved/worsened  by  the  introduction  of  the 
ONA  process. 

P 

Wed  Weekly 
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103 

1.1.18 

Measure:  Rating  of  the  organizational  behaviors  before  and  after  the  introduction  of  the  ONA 
process 

104 

1.1.18.1 

Data  Requirements:  ONA  user’s  perceptions  organizational  behaviors  before  and  after  the 
introduction  of  ONA 

105 

1.1.18.1.1 

Data  Element:  Survey  Questions  (all  7-point  Likert  Importance  Scale) 

106 

Read  each  statement  and  select  the  number  (1-7)  that  best  represents  your  response; 

107 

1 .  Attending  to  detail. 

P 

Mon  W0,  Thu  W2 

108 

2.  Relying  on  others. 

P 

Mon  W0,  Thu  W2 

109 

3.  Not  making  work  for  others. 

P 

Mon  W0,  Thu  W2 

110 

4.  Being  familiar  with  tasks  beyond  your  own  job. 

P 

Mon  W0,  Thu  W2 

111 

5.  Doing  your  job  well. 

P 

Mon  W0,  Thu  W2 

112 

6.  Working  flexibly. 

P 

Mon  W0,  Thu  W2 

113 

7.  Understanding  your  job  boundaries. 

P 

Mon  W0,  Thu  W2 

114 

8.  Understanding  the  iob  boundaries  of  others. 

P 

Mon  W0,  Thu  W2 

115 

9.  Owning  a  problem  until  it  is  resolved. 

P 

Mon  W0,  Thu  W2 

116 

1.1.19 

Measure:  Senior  Concept  Developer  perspectives  on  the  ONA  process 

117 

1.1.19.1 

Data  Requirements:  Senior  Concept  Developer  perspectives  on  the  ONA  process 

118 

1.1.19.1.1 

Data  Element:  Survey  Questions  (all  narrative  response) 

119 

Collaboration  among  organizations  within  your  nation  should  facilitate  the  collecting  of 
information  and  data  in  support  of  ONA  development. 

X 

17-Feb 

120 

Your  national  input  to  the  ONA  should  include  information  on  adversary  elements  of  power 
(PMESII)  that  was  obtained  from  a  wide  variety  of  national  organizations. 

X 

17-Feb 

121 

Through  multinational  collaboration,  the  ONA  process  identified  logical,  potential  effects  of 
friendly  actions  and  the  probable  adversary  responses  based  on  knowledge  of  ourselves  and 
of  the  adversary. 

X 

19-Feb 

122 

Through  multinational  collaboration,  the  ONA  process  identified  specific  nodes,  actions,  and 
resources  to  cause  desired  effects. 

X 

19-Feb 

123 

Through  multinational  collaboration,  the  ONA  process  effectively  and  rapidly  aggregated  and 
synthesized  data  and  information  into  coherent  knowledge  enabling  faster  and  better 

X 

19-Feb 

124 

The  ONA  process  can  improve  coalition  planning  as  compared  to  current  methods. 

X 

20-Feb 

125 

The  creation  of  a  coalition  ONA  is  viable  assuming  there  are  no  impediments  to  multinational 
information  sharing. 

X 

20-Feb 

126 

1.2 

COI:  What  are  the  impacts  of  cultural  and/or  organizational  differences  on  coalition 
collaboration? 

127 

1.2.1 

Measure:  Rating  of  impact  of  cultural  and/or  organizational  differences  on  coalition 
collaboration. 

128 

Hypothesized  response:  Cultural  and  organizational  differences  had  perceptible  impacts  on 
coalition  collaboration. 

129 

1.2. 1.1 

Data  requirement:  User  perception  of  impact  of  cultural  and/or  organizational  differences  on 
coalition  collaboration 

130 

1.2. 1.1.1 

Data  element:  Survey  questions 

131 

1 .  Cultural  differences  had  an  impact  on  the  coalition  collaboration  process.  (Impact  scale) 

X 

PC 

W2 

132 

2.  Cultural  differences  had  an  impact  on  the  quality  of  information  in  the  ONA  knowledge 

X 

PC 

W2 
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133 

3.  Organizational  differences  had  an  impact  on  the  coalition  collaboration  process.  (Impact 
scale) 

X 

PC 

W2 

134 

4.  Organizational  differences  had  an  impact  on  the  quality  of  information  in  the  ONA 
knowledge  base.  (Impact  scale) 

X 

PC 

W2 

135 

5.  Key  stakeholders  were  integral  part  of  the  planning  process. 

X 

as  occurs 

136 

1.2.2 

Measure:  Rating  of  leadership  qualities  relevant  to  assessing  coalition  collaboration  in 

MNLOE2 

137 

1.2.2. 1 

Data  Requirements:  MNLOE2  participant's  perceptions  of  leadership 

138 

1.2.2. 1.1 

Data  Element:  Survey  Questions  (all  7-point  Likert  Agree  Scale) 

139 

1.  During  MNLOE  policy  and  strategy  were  communicated  and  implemented. 

p 

Thu  W1  &  2 

140 

1.2.3 

deleted 

141 

1.2.4 

deleted 

142 

1.2.5 

Measure:  Rating  the  MNLOE2  management  practices  relevant  to  assessing  coalition 
collaboration 

143 

1.2. 5.1 

Data  Requirements:  MNLOE2  participant's  perceptions  of  MNLOE2  management  practices 

144 

1.2.5. 1.1 

Data  Element:  Survey  Questions  (all  7-point  Likert  Agree  Scale) 

145 

1.  During  MNLOE  the  ONA  business  rules  were  articulated,  understood  and  consistently 

p 

Mon  W1,TueW2 

146 

2.  During  MNLOE  communication  channels  were  open  and  effective  within  my  organization. 

p 

Mon  W1,TueW2 

147 

3.  During  MNLOE  processes  for  decision-making  allowed  for  informed  input  and  sufficient 
focus  on  issues  presented. 

p 

Mon  W1,  Tue  W2 

148 

4.  During  MNLOE  the  tools  in  support  of  planning  were  adopted  and  used. 

p 

Mon  W1 ,  Tue  W2 

149 

5.  During  MNLOE  a  risk  management  mindset  informed  decisions  about  business  practice. 

p 

Mon  W1,TueW2 

150 

6.  During  MNLOE  key  stakeholders  were  an  integral  part  of  planning  processes. 

p 

Mon  W1 ,  Tue  W2 

151 

What  tool  functions  got  in  the  way?  (Narrative) 

POC 

W1  W2 

152 

What  additional  tool  functions  would  have  been  helpful?  (Narrative) 

POC 

W1  W2 

153 

Business  rules  were  sufficiently  comprehensive  covering  all  that  was  needed  for  conducting 
the  ONA  process  (agree  scale) 

X 

POC 

W1  W2 

154 

Tools  allowed  easy  search  and  access  of  needed  information,  (agree  scale) 

P 

W1  W2 

155 

Did  you  know  when  the  ONA  knowledge  base  had  been  updated?  (Freguency  scale) 

X 

P 

W1  W2 

156 

1.2.6 

Measure:  Rating  MNLOE  systems  and  processes  relevant  to  assessing  coalition  collaboration 

157 

1.2.6. 1 

Data  Requirement:  MNLOE  participant's  perceptions  of  MNLOE  systems  and  processes 

158 

1.2.6. 1.1 

Data  Element:  Survey  Questions  (all  7-point  Likert  Agree  Scale) 

159 

1.  During  MNLOE  partnerships  across  nations  were  well  managed. 

p 

Mon  W1 ,  Tue  W2 

160 

2.  Tools  in  support  of  MNLOE  were  well  managed. 

p 

Mon  W1 ,  Tue  W2 

161 

3.  During  MNLOE  information  and  knowledge  was  well  managed. 

p 

Mon  W1,  Tue  W2 

162 

4.  Electronic  communication  systems  used  during  MNLOE  facilitated  information  sharing. 

p 

Mon  W1,  Tue  W2 

163 

5.  During  MNLOE  quality  information  was  gathered  electronically. 

p 

Mon  W1 ,  Tue  W2 

164 

6.  During  MNLOE  quality  information  was  gathered  by  non-electronic  means. 

p 

Mon  W1,  Tue  W2 

165 

7.  During  MNLOE  quality  information  was  disseminated  electronically  throughout  the 
organization. 

p 

Mon  W1,  Tue  W2 

166 

1.2.7 

Measure:  Rating  the  MNLOE  work  systems  relevant  to  assessing  coalition  collaboration 

167 

1.2. 7.1 

Data  Requirements:  MNLOE  participant's  perceptions  of  MNLOE2  work  systems 
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168 

1.2.7. 1.1 

Data  Element:  Survey  Questions  (all  7-point  Likert  Agree  Scale) 

169 

1 .  During  MNLOE  members  of  my  team  were  encouraged  to  collaborate  rather  than  compete 
with  one  another. 

P 

Thu  W1  &  2 

170 

2.  During  MNLOE  1  felt  free  to  ask  work-related  questions  of  my  team  members  without  fear  of 
being  judged. 

P 

Thu  W1  &  2 

171 

3.  During  MNLOE  1  trusted  decisions  made  by  my  leaders. 

P 

Thu  W1  &  2 

172 

4.  During  MNLOE  conflict  in  my  team  was  well  managed. 

P 

Thu  W1  &  2 

173 

5.  During  MNLOE  1  felt  free  to  discuss  my  thoughts  and  opinions  on  work  issues  with  team 
members. 

P 

Thu  W1  &  2 

174 

6.  When  1  was  not  sure  how  to  do  my  work  during  MNLOE  1  felt  comfortable  asking  others  for 
help. 

P 

Thu  W1  &  2 

175 

7.  During  MNLOE  1  was  happy  to  share  my  knowledge  and  expertise  with  all  members  of  my 

P 

Thu  W1  &  2 

176 

8.  During  MNLOE  1  was  happy  to  share  my  knowledge  and  expertise  with  other  personnel  in 
my  organization. 

P 

Thu  W1  &  2 

177 

9.  During  MNLOE  1  was  happy  to  share  my  expertise  and  knowledge  with  personnel  from 
other  countries. 

P 

Thu  W1  &  2 

178 

10.  During  MNLOE  when  1  didn't  know  how  to  do  a  particular  aspect  of  my  work  1  felt  1  needed 
to  hide  my  lack  of  knowledge. 

P 

Thu  W1  &  2 

179 

1 1 .  Please  add  any  comments  (Narrative  response) 

P 

Thu  W1  &  2 

180 

1.2.8 

Measure:  User  perception  of  impact  of  cultural  and/or  organizational  differences  on  coalition 
collaboration 

181 

1.2.8. 1 

Data  Requirements:  Participants  perceived  impacts  of  culture  and/or  organizational 
differences  on  coalition  collaboration. 

182 

1.2.8. 1.1 

Data  Element:  Survey  Questions 

183 

1.  What  cultural  differences  between  nations  do  you  think  could  i)  aid  and  ii)  hinder  national 
militaries  working  together  effectively?  (Narrative  response) 

P 

Tue  wk  0 

184 

2.  List  the  three  most  important  cultural  differences  that  had  an  impact  on  national  militaries 
working  together  effectively  during  this  week? 

P 

Thu  WO,  1,  and  2 

185 

3.  Describe  the  effects  of  one  of  these  in  detail.  (Narrative  response) 

P 

Thu  WO,  1,  and  2 

186 

4.  List  the  types  of  training  that  you  have  undertaken  in  the  past  two  years  to  understand  other 
cultures  and  overcome  cultural  differences?  (Narrative  response) 

P 

Wed  wkO 

187 

5.  List  the  (1)  formal  and  (ii)  informal  processes  that  are  in  place  in  your  national  military  to 
pass  on  experiences  with  other  national  cultures  (for  example  lessons  learned)?  (Narrative 

P 

wed  wkO 

188 

6.  The  differences  between  national  military  cultures  are  (size  scale) 

P 

Thu  WO 

189 

7.  List  the  three  most  important  organizational  differences  that  had  an  impact  on  nations 
working  together  effectively? 

P 

Thu  W2 

190 

8.  Describe  the  effects  of  one  of  these  in  detail.  (Narrative  response) 

P 

Thu  W2 

191 

1.2.9 

Measure:  User  perception  of  impact  of  the  levels  of  trust  between  national  militaries. 

192 

1.2.9. 1 

Data  Requirements:  Participants  perceived  impacts  of  the  levels  of  trust  between  national 
militaries. 

193 

1.2.9. 1.1 

Data  Element:  Survey  Questions 

194 

1 .  The  level  of  trust  between  participants  from  my  nation  had  a  significant  impact  on  the 
coalition  collaboration  process,  (agree  scale) 

P 

Thu  WO,  1,  and  2 
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195 

2.  The  level  of  trust  between  participants  from  different  nations  had  a  significant  impact  on  the 
coalition  collaboration  process,  (agree  scale) 

P 

Thu  W0,  1,  and  2 

196 

3. The  level  of  trust  required  between  participants  from  different  nations  for  an  effective 
coalition  collaboration  process  was  (deqree  scale) 

P 

Thu  W2 

197 

4.  How  did  the  level  of  trust  between  participants  from  different  nations  (i)  aid  or  (ii)  hinder 
them  working  together  effectively?  (Narrative  response) 

P 

Thu  W0,  1,  and  2 

198 

1.2.10 

Measure  User  perception  of  the  impact  of  levels  of  training  and  experience  on  the  coalition 
collaboration  process 

199 

1.2.10.1 

Data  Requirement:  Player  perception  of  training  on  collaboration  process 

200 

1.2.10.1.1 

Data  Element:  Survey  questions 

201 

1 .  As  of  today,  how  familiar  are  you  with  the  ONA  process?  (Familiar  scale) 

P 

Daily 

202 

2.  As  of  today,  how  comfortable  are  you  with  the  tools  used  during  the  experiment?  (Comfort 
scale) 

P 

Daily 

203 

1 .  Are  there  differences  in  the  levels  of  training  (relevant  to  this  exercise)  between 
participating  national  militaries?  (Yes/No) 

P 

Thu  W2 

204 

2.  Differences  in  levels  of  training  between  participating  national  militaries  had  a  significant 
impact  on  the  coalition  collaboration  process.  (Agree  scale) 

P 

Thu  W2 

205 

3.  Are  there  differences  between  participating  national  militaries  in  relation  to  training 
priorities  ?  Yes/No 

P 

Thu  W2 

206 

4.  Differences  in  training  priorities  between  national  militaries  had  a  significant  impact  on  the 
coalition  collaboration  process.  (Agree  scale) 

P 

Thu  W2 

207 

5.  Are  there  differences  in  the  levels  of  experience  (relevant  to  this  exercise)  between 
participating  national  militaries?  Yes/No. 

P 

Thu  W2 

208 

6.  Differences  in  levels  of  experience  between  participating  national  militaries  had  a  significant 
impact  on  the  coalition  collaboration  process.  (Agree  scale) 

P 

Thu  W2 

209 

7.  How  familiar  were  you  with  the  ONA  process  before  this  exercise?  (familiar  scale) 

P 

Mon  W1 

210 

8.  What  additional  training  and/or  experience  would  have  helped  you  to  carry  out  your  tasks 
during  this  exercise?  (Narrative) 

P 

Thu  W2 

211 

9.  Interactions  include  combined  exercises,  exchange  visits,  social  interaction  and  any  other 
form  of  interaction  (please  specify).  Which  of  the  these  forms  of  interaction  had  the  greatest 
impact  on  your  ability  to  work  effectively  with  other  national  militaries?  (Narrative  response) 

P 

Thu  W2 

212 

10.  Briefly  describe  how.  (Narrative  response) 

P 

Thu  W2 

213 

1.2.11 

Measure:  User  perception  of  degree  of  difficulty  of  fitting  into  the  command  structure. 

214 

1.2.11.1 

Data  Requirement:  Player  perception  of  degree  of  difficulty  of  fitting  into  the  command 

215 

1.2.11.1.1 

Data  Element:  Survey  questions 

216 

1 .  Compared  to  what  1  am  used  to,  the  expectations  of  superiors  of  other  nationalities  were 
(same  scale) 

P 

Thu  W2 

217 

2.  Compared  to  what  1  am  used  to,  the  command  style  of  superiors  of  other  nationalities  was 
(same  scale) 

P 

Thu  W2 

218 

3.  Superiors  of  other  nationalities  had  a  clear  understanding  of  my  capabilities  (agree  scale) 

P 

Thu  W0,  1  and  2 

219 

4.  Superiors  of  other  nationalities  were  responsive  to  my  needs  (agree  scale). 

P 

Thu  W0,  1  and  2 

220 

5.  My  responsibilities  during  this  exercise  were  (clear  scale) 

P 

Thu  W0,  1  and  2 
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221 

6.  Superiors  of  other  nationalities  anticipated  my  needs,  (agree  scale) 

P 

Thu  W0,  1  and  2 

222 

7. 1  had  sufficient  authority  to  carry  out  my  task,  (agree  scale) 

P 

Thu  W0,  1  and  2 

223 

8.  What  problems  did  you  experience  with  the  command  and  control  structure  within  which  you 
were  working  during  this  week?  (Narrative  response) 

P 

Thu  W0,  1  and  2 

224 

1.2.12 

Measure:  User  perception  of  sharing  of  information  and  differences  in  participants 
understanding. 

225 

1.2.12.1 

Data  Requirement:  Player  perception  of  sharing  of  information  and  differences  in  participants 
understanding. 

226 

1.2.12.1.1 

Data  Element:  Survey  questions 

227 

1 . 1  understood  the  instructions  given  (agree  scale) 

P 

Thu  W0,  1  and  2 

228 

2.  1  understood  the  terms  used  (agree  scale) 

P 

Thu  W0,  1  and  2 

229 

3. 1  understood  the  communications  received  (agree  scale) 

P 

Thu  W0,  1  and  2 

230 

4.  Participants  from  other  nations  understood  the  instructions  given  (agree  scale) 

P 

Thu  W0,  1  and  2 

231 

5.  Participants  from  other  nations  understood  the  terms  used  (agree  scale) 

P 

Thu  W0,  1  and  2 

232 

6.  Participants  from  other  nations  understood  the  communications  received  (agree  scale) 

P 

Thu  W0,  1  and  2 

233 

7.  Participants  from  other  nations  understood  my  point  of  view  (agree  scale) 

P 

Thu  W0,  1  and  2 

234 

8.  The  information  provided  via  the  shared  tools  was  reliable,  (agree  scale) 

P 

Thu  W0,  1  and  2 

235 

9.  The  information  provided  via  the  shared  tools  was  relevant,  (agree  scale) 

P 

Thu  W0,  1  and  2 

236 

10.  The  information  provided  via  the  shared  tools  was  easy  to  understand  (agree  scale) 

P 

Thu  W0,  1  and  2 

237 

11.1  had  access  to  all  of  the  information  required  to  carry  out  my  task  (agree  scale) 

P 

Thu  W0,  1  and  2 

238 

12.  What  could  be  done  to  increase  the  understanding  of  information,  instructions  and/or 
communications  between  participating  national  militaries?  (Narrative  response) 

P 

Thu  W0,  1  and  2 

239 

1.2.13 

Measure:  Senior  Concept  Developer  perspectives  on  the  impact  of  cultural  and/or 
organizational  differences 

240 

1.2.13.1 

Data  Requirements:  Senior  Concept  Developer  perspectives  on  the  impact  of  cultural  and/or 
organizational  differences 

241 

1.21.13.1.1 

Data  Element:  Survey  Questions  (all  narrative  response) 

242 

The  political  culture  of  your  nation  is  compatible  with  the  requirements  of  the  ONA  process. 

X 

18-Feb 

243 

The  military  culture  of  your  nation  is  compatible  with  the  requirements  of  the  ONA  process. 

X 

18-Feb 

244 

The  social  customs  of  your  nation  are  compatible  with  the  requirements  of  the  ONA  process. 

X 

18-Feb 

245 

The  political  culture  of  your  nation  is  compatible  with  the  requirements  of  sharing  information 
that  is  required  to  develop  a  coalition  ONA. 

X 

24-Feb 

246 

The  military  culture  of  your  nation  is  compatible  with  the  requirements  of  sharing  information 
that  is  required  to  develop  a  coalition  ONA. 

X 

24-Feb 

247 

The  social  customs  of  your  nation  are  compatible  with  the  requirements  of  sharing  information 
that  is  required  to  develop  a  coalition  ONA. 

X 

24-Feb 

248 

2.0 

Objective:  Identify  and  assess  issues  associated  with  collaboration  and  information 
sharing  across  different  security  domains 

249 

2.1 

COI:  Does  collaboration  with  coalition  partners  across  security  domains  improve  the 
ONA? 

250 

2.1.1 

Measure:  Percent  of  data  in  the  ONA  knowledge  base  available  to  participants 
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251 

Hypothesized  response:  The  percent  of  data  in  the  ONA  knowledge  base  available  to 
participants  will  increase  over  time  during  an  individual  trial  week,  and  will  further  increase 
comparing  the  future  to  the  current  method  of  information  sharing  trials. 

252 

2.1. 1.1 

Data  requirement:  Availability  of  ONA  data  to  participants  at  each  Coalition  network  node  over 
time 

253 

2.1. 1.1.1 

Total  number  of  data  records  available  in  the  ONA  knowledge  base 

X 

As  Occurring 

254 

2.1. 1.1.2 

Number  of  node  data  records  available  in  the  ONA  knowledge  base 

X 

As  Occurring 

255 

2.1. 1.1.3 

Number  of  effect  data  records  available  in  the  ONA  knowledge  base 

X 

As  Occurring 

256 

2.1. 1.1.4 

Number  of  action  data  records  available  in  the  ONA  knowledge  base 

X 

As  Occurring 

257 

2.1. 1.1. 5 

Number  of  resource  data  records  available  in  the  ONA  knowledge  base 

X 

As  Occurring 

258 

2.1. 1.1.6 

Number  of  data  records  originator  has  identified  as  critical  available  in  the  ONA  knowledge 

X 

As  Occurring 

259 

2.1. 1.1. 7 

Total  number  of  data  records  available  to  each  participant  in  the  Coalition  Network 

X 

As  Occurring 

260 

2.1. 1.1.8 

Number  of  node  data  records  available  to  each  participant  in  the  Coalition  Network 

X 

As  Occurring 

261 

2.1. 1.1.9 

Number  of  effect  data  records  available  to  each  participant  in  the  Coalition  Network 

X 

As  Occurring 

262 

2.1.1.1.10 

Number  of  action  data  records  available  to  each  participant  in  the  Coalition  Network 

X 

As  Occurring 

263 

2.1.1.1.11 

Number  of  resource  data  records  available  to  each  participant  in  the  Coalition  Network 

X 

As  Occurring 

264 

2.1.1.1.12 

Number  of  data  records  originator  has  identified  as  critical  available  to  each  participant  in  the 
Coalition  Network 

X 

As  Occurring 

265 

2.1.2 

Measure:  Rating  of  situational  awareness  of  each  participant  in  the  Coalition  Network  over 

266 

Hypothesized  response:  Participant  situational  awareness  will  increase  over  time  during  an 
individual  trial  week,  and  will  further  increase  comparing  the  future  to  the  current  method  of 
information  sharing  trials. 

267 

2.1.2. 1 

Data  requirement:  Comparison  of  user  perception  with  ground  truth  data. 

268 

2.1.2. 1.1 

Ground  truth  data  for  Probe  Questions.  (Used  again  on  375) 

X 

Three  Times  Daily 

269 

2.1.2. 1.2 

Data  element:  Survey  questions 

270 

1 .  How  do  you  rate  your  overall  situational  awareness?  (quality  scale) 

P 

Daily 

271 

2.  Detailed  questions  on  situations  that  should  be  known  to  the  participants  at  each  survey 
interval,  (multiple  choice)  (Used  again  on  375) 

PC 

Three  Times  Daily 

272 

2.1.3 

Measure:  Rating  of  ONA  quality  when  data  is  shared  across  security  domains 

273 

Hypothesized  response:  Participants  will  perceive  an  improvement  in  ONA  quality  when  the 
future  information  sharing  trial  is  compared  to  the  current  method  of  information  sharing  trial. 

274 

2.1 .3.1 

Data  requirement:  ONA  users  perception  of  value  of  collaboration  across  security  domains 
compared  to  current  methods 

275 

2.1. 3.1.1 

Data  element:  Survey  questions 

276 

1 .  Information  sharing  across  security  domains  enhanced  the  quality  of  information  in  the 

ONA  knowledge  base,  (agree  scale) 

COP 

W1  W2 

277 

2.  The  ONA  knowledge  base  supported  understanding  of  Blue’s  goals,  intentions,  strengths 
and  weaknesses,  (aqree  scale) 

COP 

W2 

278 

3.  The  ONA  knowledge  base  supported  understanding  of  Red's  goals,  intentions,  strengths 
and  weaknesses,  (aqree  scale) 

COP 

W2 

279 

4.  Information  1  needed  was  in  the  ONA  knowledge  base.  (Frequency  Scale) 

p 

W1  W2 
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280 

5.  To  what  degree  did  Information  Sharing  Procedures  allow  you  to  share  information  with 
other  members  during  the  ONA  development  process?  (degree  scale) 

P 

W1  W2 

281 

6.  To  what  degree  did  Information  Sharing  Procedures  allow  other  members  of  the  ONA 
development  process  to  share  information  with  you?  (degree  scale) 

P 

W1  W2 

282 

2.1.4 

Measure:  Instances  of  releasability  violations. 

283 

Hypothesized  response:  Releasability  violations  from  the  ONA  knowledge  base  will  decrease 
as  participants  become  more  familiar  with  disclosure  and  release  policies  and  procedures. 

284 

2.1.4. 1 

Data  requirement:  Instances  of  releasability  violations  from  ONA  knowledge  base  over  time 

285 

2.1.4. 1.1 

Data  element:  Survey  questions 

286 

Were  there  any  releasability  violations  today?  (Yes/No)  If  so,  describe. 

X 

P 

Daily 

287 

2.1.5 

deleted 

288 

2.1.6 

deleted 

289 

2.1.7 

Measure:  Rating  of  Week  1  and  2  differences  in  collaboration  and  information  sharing 

290 

Hypothesized  response:  Participants  will  perceive  an  improvement  in  collaboration  when  the 
future  information  sharing  trial  is  compared  to  the  current  method  of  information  sharing  trial. 

291 

2.1.7. 1 

Data  Requirement:  User  perception  of  differences  between  Week  1  and  Week  2  of 
collaboration  with  coalition  partners  across  security  domains  in  the  ONA  process. 

292 

2.1.7. 1.1 

Data  Element:  Survey  questions 

293 

1.  What  were  the  top  three  differences  in  collaboration  between  Week  1  and  Week  2? 
(Narrative  response) 

POC 

W2 

294 

2.  What  were  the  top  three  differences  in  information  sharing  between  Week  1  and  Week  2? 
(Narrative  response) 

POC 

W2 

295 

2.1.8 

Measure:  Rating  of  satisfaction  with  the  ONA  process 

296 

2.1.8. 1 

Data  Requirement:  ONA  user's  perception  of  their  satisfaction  with  the  ONA  process 

297 

2.1 .8.1.1 

Data  Element:  Survey  Questions  (all  7-point  Likert  Satisfaction  Scale) 

298 

1 .  How  do  you  feel  about  the  precision  of  the  information  the  ONA  process  provides? 

p 

Wed  Weekly 

299 

2.  How  do  you  feel  about  the  information  content  provided  by  the  ONA  process? 

p 

Wed  Weekly 

300 

3.  How  do  you  feel  about  how  the  information  provided  by  the  ONA  process  meets  your 

p 

Wed  Weekly 

301 

4.  How  do  you  feel  about  the  sufficiency  of  the  information  provided  by  the  ONA  process? 

p 

Wed  Weekly 

302 

5.  How  do  you  feel  about  the  accuracy  of  the  information  provided  by  the  ONA  process? 

p 

Wed  Weekly 

303 

6.  How  satisfied  are  you  with  the  accuracy  of  the  ONA  process? 

p 

Wed  Weekly 

304 

7.  How  do  you  feel  about  the  usefulness  of  the  output  from  the  ONA  process? 

p 

Wed  Weekly 

305 

8.  How  do  you  feel  about  the  clarity  of  the  information  produced  by  the  ONA  process? 

p 

Wed  Weekly 

306 

9.  How  do  you  feel  about  the  timeliness  of  the  information  provided  by  the  ONA  process? 

p 

Wed  Weekly 

307 

10.  How  do  you  feel  about  the  currency  of  the  information  provided  by  the  ONA  process? 

p 

Wed  Weekly 

308 

2.1.9 

Measure:  Senior  Concept  Developer  perspectives  on  collaboration  and  information  sharing 

309 

2.1 .9.1 

Data  Requirements:  Senior  Concept  Developer  perspectives  on  collaboration  and  information 
sharing 

310 

2.1. 9.1.1 

Data  Element:  Survey  Questions  (all  narrative  response) 

311 

Your  nation's  diplomatic,  information,  military  and  economic  (DIME)  concerns  were 
satisfactorily  addressed  through  the  use  of  the  collaborative  process. 

X 

25-Feb 
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312 

Information  and  data  that  was  most  critical  to  the  development  of  the  multinational  ONA  was 
shared  by  nations  in  a  satisfactory  manner. 

X 

26-Feb 

313 

Multinational  information  sharing  effectively  supported  the  ONA  process. 

X 

26-Feb 

314 

Multinational  information  sharing  was  conducted  in  a  timely  manner. 

X 

27-Feb 

315 

The  Future  Multinational  Information  Sharing  (MNIS)  method  improved  multinational 
information  sharing  as  compared  to  current  methods. 

X 

27-Feb 

316 

2.2 

COI:  What  impact  does  information  sharing  have  on  each  phase  of  the  ONA  process? 

317 

2.2.1 

Measure:  Rating  of  impact  of  information  sharing  on  identification  of  Potential  Effects 

318 

Hypothesized  response:  Participants  will  perceive  that  information  sharing  enhances  the 
quality  of  potential  effects  identified. 

319 

2.2. 1.1 

Data  requirement:  User  perception  of  impact  of  information  sharing  on  identification  of 

Potential  Effects 

320 

2.2. 1.1.1 

Data  element:  Survey  questions 

321 

In  most  cases,  the  degree  to  which  the  coalition  conducted  Wargaming  was  sufficient  to 
develop  good-quality  Effects,  (agree  scale) 

X 

PC 

W1  W2 

322 

How  often  did  different  national  perspectives  enhance  the  development  of  Effects?  (frequency 
5  pt) 

X 

PC 

W1  W2 

323 

Did  your  nation  make  compromises  about  Effects  because  of  opposition  by  the  coalition?  If  so, 
what  kinds  of  compromises  did  your  nation  make?  (narrative) 

X 

PC 

W1  W2 

324 

What,  if  any,  were  the  sources  of  disagreement  among  nations  concerning  Effects?  (narrative) 

X 

PC 

W1  W2 

325 

How  often  could  your  nation  have  proposed  additional  effects  to  the  coalition,  but  did  not  in 
order  to  protect  information?  (frequency  5  pt) 

X 

PC 

W1  W2 

326 

How  often  did  Future  information  sharing  enhance  the  range  of  Effects  available  to  the 
coalition?  (frequency  5  pt) 

X 

PC 

W2 

327 

2.2.2 

Measure:  Frequency  of  ONA  knowledge  base  effect  record  access 

328 

Hypothesized  response:  Effect  record  access  may  change  as  participant's  understanding  of 
the  ONA  process  improves. 

329 

2.2.2. 1 

Data  requirement:  Number  of  knowledge  base  effect  record  accesses  by  each  participant  in 
the  coalition  network  over  time 

330 

2.2.2. 1.1 

Number  of  knowledge  base  effect  record  accesses  by  each  participant  in  the  coalition  network 

X 

Hourly 

331 

2.2.3 

Measure:  Frequency  of  ONA  knowledge  base  effect  record  update 

332 

Hypothesized  response:  Effect  record  update  may  change  as  participant's  understanding  of 
the  ONA  process  improves. 

333 

2.2.3. 1 

Data  requirement:  Number  of  knowledge  base  effect  record  updates  by  each  participant  in  the 
coalition  network  over  time 

334 

2.2.3. 1.1 

Number  of  knowledge  base  effect  record  updates  by  each  participant  in  the  coalition  network 

X 

Hourly 

335 

2.2.4 

Measure:  Rating  of  impact  of  information  sharing  on  identification  of  Nodes 

336 

deleted 

337 

2.2.5 

Measure:  Frequency  of  ONA  knowledge  base  node  record  access 

338 

Hypothesized  response:  Node  record  access  may  change  as  participant's  understanding  of 
the  ONA  process  improves. 
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339 

2.2.5. 1 

Data  requirement:  Number  of  knowledge  base  node  record  accesses  by  each  participant  in 
the  coalition  network  over  time 

340 

2.2.5. 1.1 

Number  of  knowledge  base  node  record  accesses  by  each  participant  in  the  coalition  network 

X 

Hourly 

341 

2.2.6 

Measure:  Frequency  of  ONA  knowledge  base  node  record  update 

342 

Hypothesized  response:  Node  record  update  may  change  as  participant's  understanding  of 
the  ONA  process  improves. 

343 

2.2.6. 1 

Data  requirement:  Number  of  knowledge  base  node  record  updates  by  each  participant  in  the 
coalition  network  over  time 

344 

2.2.6. 1.1 

Number  of  knowledge  base  node  record  updates  by  each  participant  in  the  coalition  network 

X 

Hourly 

345 

2.2.7 

Measure:  Rating  of  impact  of  information  sharing  on  Linkage  of  Effects  to  Nodes 

346 

Hypothesized  response:  Participants  will  perceive  that  information  sharing  enhances  the 
quality  of  effect  to  node  linkages  identified. 

347 

2.2.7. 1 

Data  requirement:  User  perception  of  impact  of  information  sharing  on  Linkage  of  effects  to 
Nodes 

348 

2.2.7. 1.1 

Data  element:  Survey  questions 

349 

How  often  were  Effects  sufficiently  focused  to  allow  satisfactory  Effect-to-Node  linkages? 

X 

PC 

W1  W2 

350 

In  most  cases,  were  descriptions  of  Nodes  either  too  specific  or  too  general  to  allow 
satisfactory  linkage  to  Effects?  (detail  scale) 

X 

PC 

W1  W2 

351 

How  often  did  different  national  perspectives  enhance  Effect-to-Node  linkages?  (frequency  5 

X 

PC 

W1  W2 

352 

What,  if  any,  were  the  differences  in  how  nations  approached  Effect-to-Node  linkages? 

X 

PC 

W1  W2 

353 

What,  if  any,  were  the  sources  of  disagreement  among  nations  concerning  Effect-to-Node 
linkages?  (narrative) 

X 

PC 

W1  W2 

354 

How  often  did  your  nation  refrain  from  proposing  additional  Effect-to-Node  linkages  to  the 
coalition,  in  order  to  protect  information?  (frequency  5  pt) 

X 

PC 

W1  W2 

355 

How  often  did  Future  information  sharing  increase  the  range  of  Effect-to-Node  linkages 
available  to  the  coalition?  (frequency  5  pt) 

X 

PC 

W2 

356 

2.2.8 

Measure:  Frequency  of  ONA  knowledge  base  linkage  record  access 

357 

Hypothesized  response:  Linkage  record  access  may  change  as  participant's  understanding  of 
the  ONA  process  improves. 

358 

2.2.8. 1 

Data  requirement:  Number  of  knowledge  base  linkage  record  accesses  by  each  participant  in 
the  coalition  network  over  time 

359 

2.2.8. 1.1 

Number  of  knowledge  base  linkage  record  accesses  by  each  participant  in  the  coalition 

X 

Hourly 

360 

2.2.9 

Measure:  Frequency  of  ONA  knowledge  base  linkage  record  update 

361 

Hypothesized  response:  Linkage  record  update  may  change  as  participant's  understanding  of 
the  ONA  process  improves. 

362 

2.2.9. 1 

Data  requirement:  Number  of  knowledge  base  linkage  record  updates  by  each  participant  in 
the  coalition  network  over  time 

363 

2.2.9. 1.1 

Number  of  knowledge  base  linkage  record  updates  by  each  participant  in  the  coalition  network 

X 

Hourly 

364 

2.2.10 

Measure:  Rating  of  impact  of  information  sharing  on  selection  of  DIME  options 

365 

deleted 

366 

2.2.11 

Measure:  Frequency  of  ONA  knowledge  base  DIME  option  record  access 

367 

Hypothesized  response:  DIME  option  record  access  may  change  as  participant's 
understanding  of  the  ONA  process  improves. 
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368 

2.2.11.1 

Data  requirement:  Number  of  knowledge  base  DIME  option  record  accesses  by  each 
participant  in  the  coalition  network  overtime 

369 

2.2.11.1.1 

Number  of  knowledge  base  DIME  option  record  accesses  by  each  participant  in  the  coalition 
network 

X 

Hourly 

370 

2.2.12 

Measure:  Frequency  of  ONA  knowledge  base  DIME  option  record  update 

371 

Hypothesized  response:  DIME  option  record  update  may  change  as  participant's 
understanding  of  the  ONA  process  improves. 

372 

2.2.12.1 

Data  requirement:  Number  of  knowledge  base  DIME  option  record  updates  by  each 
participant  in  the  coalition  network  overtime 

373 

2.2.12.1.1 

Number  of  knowledge  base  DIME  option  record  updates  by  each  participant  in  the  coalition 
network 

X 

Hourly 

374 

2.2.13 

Measure:  Rating  of  impact  of  information  sharing  on  identification  of  2nd  and  3rd  order  effects 

375 

deleted 

376 

2.2.14 

Measure:  Frequency  of  ONA  knowledge  base  2nd  and  3rd  order  effect  record  access 

377 

Hypothesized  response:  2nd  and  3rd  order  effect  record  access  may  change  as  participant's 
understanding  of  the  ONA  process  improves. 

378 

2.2.14.1 

Data  requirement:  Number  of  knowledge  base  2nd  and  3rd  order  effect  record  accesses  by 
each  participant  in  the  coalition  network  over  time 

379 

2.2.14.1.1 

Number  of  knowledge  base  2nd  and  3rd  order  effect  record  accesses  by  each  participant  in 
the  coalition  network 

X 

Hourly 

380 

2.2.15 

Measure:  Frequency  of  ONA  knowledge  base  2nd  and  3rd  order  effect  record  update 

381 

Hypothesized  response:  2nd  and  3rd  order  effect  record  update  may  change  as  participant's 
understanding  of  the  ONA  process  improves. 

382 

2.2.15.1 

Data  requirement:  Number  of  knowledge  base  2nd  and  3rd  order  effect  record  updates  by 
each  participant  in  the  coalition  network  over  time 

383 

2.2.15.1.1 

Number  of  knowledge  base  2nd  and  3rd  order  effect  record  updates  by  each  participant  in  the 
coalition  network 

X 

Hourly 

384 

2.2.16 

Measure:  Ratinq  of  impact  of  information  sharinq  on  selection  of  resources 

385 

deleted 

386 

2.2.17 

Measure:  Frequency  of  ONA  knowledge  base  resource  record  access 

387 

Hypothesized  response:  Resource  record  access  may  change  as  participant's  understanding 
of  the  ONA  process  improves. 

388 

2.2.17.1 

Data  requirement:  Number  of  knowledge  base  resource  record  accesses  by  each  participant 
in  the  coalition  network  over  time 

389 

2.2.17.1.1 

Number  of  knowledge  base  resource  record  accesses  by  each  participant  in  the  coalition 
network 

X 

Hourly 

390 

2.2.18 

Measure:  Frequency  of  ONA  knowledge  base  resource  record  update 

391 

Hypothesized  response:  Resource  record  update  may  change  as  participant's  understanding 
of  the  ONA  process  improves. 

392 

2.2.18.1 

Data  requirement:  Number  of  knowledge  base  resource  record  updates  by  each  participant  in 
the  coalition  network  over  time 

393 

2.2.18.1.1 

Number  of  knowledge  base  resource  record  updates  by  each  participant  in  the  coalition 

X 

Hourly 

394 

2.2.19 

Measure:  Rating  of  impact  of  information  sharing  on  identification  of  actions 
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395 

Hypothesized  response:  Participants  will  perceive  that  information  sharing  enhances  the 
quality  of  actions  identified. 

396 

2.2.19.1 

Data  requirement:  User  perception  of  impact  of  information  sharinq  on  identification  of  actions 

397 

2.2.19.1.1 

Data  element:  Survey  questions 

398 

How  often  were  Effect-to-Node  linkages  defined  clearly  enough  to  easily  see  the  advantages 
of  Action  alternatives?  (frequency  5  pt) 

X 

PC 

W1  W2 

399 

How  often  did  different  national  perspectives  enhance  the  range  of  Actions  available  to  the 
coalition?  (frequency  5  pt) 

X 

PC 

W1  W2 

400 

What,  if  any,  were  the  sources  of  disagreement  among  nations  concerning  Actions? 

X 

PC 

W1  W2 

401 

How  often  did  your  nation  refrain  from  proposing  additional  Actions  to  the  coalition,  in  order  to 
protect  information?  (frequency  5  pt) 

X 

PC 

W1  W2 

402 

How  often  did  Future  information  sharing  increase  the  range  of  Actions  available  to  the 
coalition?  (frequency  5  pt) 

X 

PC 

W1  W2 

403 

2.2.20 

Measure:  Rating  of  impact  of  information  sharinq  on  view  of  Blue  and  Red 

404 

Hypothesized  response:  Participants  will  perceive  that  information  sharing  enhances  the 
quality  of  view  of  Blue  and  Red 

405 

2.2.20.1 

Data  requirement:  User  perception  of  impact  of  information  sharing  on  view  of  Blue  and  Red 

406 

2.2.20.1.1 

Data  element:  Survey  questions 

407 

In  most  cases,  coalition  objectives  were  sufficiently  focused  to  allow  useful  views  of  Blue  and 
Red  to  be  development,  (agree  scale) 

X 

PC 

W1  W2 

408 

To  what  degree  did  different  national  perspectives  enhance  coalition  views  of  Blue  and  Red? 
(degree  scale) 

X 

PC 

W1  W2 

409 

What,  if  any,  were  the  sources  of  disagreement  among  nations  concerning  views  of  Blue  and 
Red?  (narrative) 

X 

PC 

W1  W2 

410 

How  often  could  your  nation  have  altered  coalition  views  of  Blue  and  Red,  but  did  not  in  order 
to  protect  information?  (frequency  5pt) 

X 

PC 

W1  W2 

411 

2.2.21 

Measure:  Rating  of  impact  of  information  sharinq  on  Warqaminq 

412 

Hypothesized  response:  Participants  will  perceive  that  information  sharing  enhances  the 
quality  of  Warqaminq 

413 

2.2.21.1 

Data  requirement:  User  perception  of  impact  of  information  sharing  on  Wargaming 

414 

2.2.21.1.1 

Data  element:  Survey  questions 

415 

Coalition  views  of  Blue  and  Red  were  comprehensive  enough  for  a  thorough  exchange  of 
challenges  and  counter-actions  during  the  Warqaminq  process,  (agree  scale) 

X 

PC 

W1  W2 

416 

How  often  did  different  national  perspectives  enhance  the  Wargaming  process?  (frequency 

X 

PC 

W1  W2 

417 

2.3 

COI:  Does  the  MNIS  future  operational  concept  speed  up  the  ONA  process? 

418 

2.3.1 

deleted 

419 

2.3.2 

Measure:  Rate  of  change  in  data  records  available  to  each  participant  in  the  Coalition  Network 

420 

Hypothesized  response:  The  future  MNIS  methods  will  increase  the  rate  that  data  in  the  ONA 
knowledge  base  becomes  available  to  participants  compared  to  current  information  sharing 
methods. 

421 

2.3.2. 1 

Data  requirement:  Availability  of  data  to  ONA  users  at  each  Coalition  node  over  time 

422 

2.3.2. 1.1 

Same  data  as  measure  2.1.1 

X 

Hourly 
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423 

2.3.3 

Measure:  Rating  of  situational  understanding  of  each  participant  in  the  Coalition  Network 
(same  as  measure  2.1.2) 

424 

Hypothesized  response:  Participant  situational  awareness  will  increase  over  time  during  an 
individual  trial  week,  and  will  further  increase  when  the  future  information  sharing  trial  is 
compared  to  the  current  method  of  information  sharing  trial. 

425 

2.3.3. 1 

Data  requirement:  Comparison  of  user  perception  with  ground  truth  data  (Same  Ground  truth 
data  and  Probe  Questions  as  under  2.1.2) 

426 

2.3.4 

Measure:  Rating  of  difference  between  Week  1  and  Week  2  of  the  MNIS  operational  concept 
on  the  speed  of  the  ONA  process. 

427 

Hypothesized  response:  The  future  MNIS  operational  concept  will  improve  the  speed  of  the 
ONA  process. 

428 

2.3.4. 1 

Data  requirement:  ONA  users  perception  of  adequacy  of  future  OPCON  in  speeding  up  the 
ONA  process 

429 

2.3.4. 1.1 

Data  element:  Survey  questions 

430 

1 .  Did  Week  2  information  sharing  procedures  speed  up  the  ONA  process?  (yes/no)  Please 
describe. 

POC 

W2 

431 

2.4 

COI:  Can  information  releasability  procedures  keep  the  ONA  data  current? 

432 

2.4.1 

deleted 

433 

2.4.2 

deleted 

434 

2.4.3 

Measure:  Rating  of  difference  between  Week  1  and  Week  2  of  ability  of  information 
releasability  procedures  to  keep  the  ONA  data  current 

435 

Hypothesized  response:  The  future  MNIS  information  releasability  procedures  will  improve  the 
ability  to  keep  ONA  data  current. 

436 

2.4.3. 1 

Data  Requirement:  User  perception  of  difference  between  Week  1  and  Week  2  of  ability  of 
information  releasability  procedures  to  keep  the  ONA  data  current 

437 

2.4.3. 1.1 

Data  Element:  Survey  questions 

438 

1 .  Information  releasability  procedures  were  able  to  keep  the  ONA  knowledge  base  current, 
(agree  scale) 

COP 

W1  W2 

439 

2.  Which  information  release  procedures  inhibited  your  ability  to  keep  the  ONA  current? 
(Narrative  response) 

COP 

W1  W2 

440 

3.  Which  information  release  procedures  facilitated  your  ability  to  keep  the  ONA  current? 
(Narrative  response) 

COP 

W1  W2 
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APPENDIX  G.  MN  LOE  II  DEMOGRAPHICS 


A  demographic  survey  was  administered  at  the  end  of  Week  0  to  all  experiment 
participants,  as  well  as  to  selected  controllers  and  observers,  to  collect  information  on  the 
participants’  backgrounds,  experience,  and  familiarity  with  the  experiment  concepts.  Of 
the  78  respondents,  51  were  planners  and  SoSAs,  and  27  were  observers  and  controllers. 

This  bar  chart  depicts  the  nationalities  of  those  surveyed,  as  well  as  their 
experiment  site.  The  smaller  bars  seen  under  GER,  UK,  and  US  indicate  the  breakdown 
of  the  nationalities  for  the  regional  partner,  NATO.  Two  participants  indicated  a 
nationality  of  “other”:  One  was  a  Polish  officer  within  NATO,  and  one  was  a  participant 
with  dual  citizenship  with  Australia  and  the  United  Kingdom. 


These  experienced  participants  included  a  large  contingent  of  contractors  (33 
percent  of  total  respondents,  72  percent  of  U.S.  respondents).  Of  the  26  contractors  who 
responded,  21  had  prior  military  experience.  Of  the  11  civilian  government  employees, 
six  also  had  prior  military  experience.  In  fact,  74  percent  of  the  respondents  had  more 
than  10  years  of  military  experience.  Of  those  with  military  experience,  76  percent 
specialized  in  combat  arms  or  intelligence. 
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The  experiment  participants  were  also  well  educated.  Of  the  78  respondents,  95 
percent  held  at  least  a  BA/BS  degree  or  equivalent,  and  67  percent  held  a  master’s  degree 
or  PhD. 
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The  two  primary  objectives  of  this  experiment  involved  (1)  examination  of  the 
ability  of  national  headquarters  staffs  to  conduct  a  distributed  ONA,  and  (2)  multinational 
collaborative  information  sharing  across  different  security  domains.  To  this  end, 
participants  needed  a  familiarity  with  distributed  teamwork,  prior  exposure  to  the  Groove 
collaboration  tool,  and  prior  exposure  to  the  U.S.  ONA  concept. 


More  than  79  percent  of  the  participants  rated  themselves  as  familiar  or  very 
familiar  with  distributed  teamwork.  Only  33  percent  had  moderate  or  significant  exposure 
to  Groove  prior  to  the  experiment;  37  percent  had  moderate  or  significant  pre-experiment 
exposure  to  the  U.S.  ONA  concept.  Only  17  percent  of  the  non-U. S.  participants  had 
moderate  or  significant  prior  exposure  to  the  concept. 
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Participant  preparation  may  include  previous  joint  and  combined  experience,  as  well 
as  experience  with  the  partner  nations  in  exercises,  exchange  visits,  and  social  interactions. 

Almost  77  percent  of  the  respondents  had  one  or  more  years  of  joint  experience, 
and  68  percent  had  one  or  more  years  of  multinational  or  combined  experience. 

The  following  charts  depict  the  respondents’  experience  in  combined  exercises, 
exchange  visits,  and  social  interactions  with  each  of  the  partner  nations. 
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APPENDIX  H.  ANCILLARY  LESSONS  LEARNED 


The  lessons  learned  were  categorized  as  main  experimental  lessons  learned  and 
ancillary  lessons  learned.  The  main  lessons  learned  aimed  to  facilitate  the  successful 
completion  of  the  next  multinational  experiment.  After  those  lessons  are  implemented,  or 
possibly  at  the  same  time,  the  ancillary  lessons  learned  should  be  implemented  if  time  and 
resources  permit.  See  Chapter  V,  Experimental  Process  Lessons,  for  more  information. 

Lessons  learned  are  presented  in  this  fonnat: 

Title  of  Lesson  Learned: 

□  Observation:  This  is  the  point  of  concern. 

□  Discussion:  The  discussion  expounds  upon  the  observation  and  gives  more 
background  into  the  potential  problem  or  solution. 

□  Lessons  Learned:  What  was  learned  from  the  observation  that  was  made? 

□  Recommendation:  What  is  the  recommendation  for  future  experimentation? 

TECHNOLOGY 

Title:  Network  Server  Location 

□  Observation:  Servers  were  not  positioned  to  provide  optimum  service. 

□  Discussion:  The  U.S.  Groove  Relay  Server  and  the  ONADB  Server  were  located 
on  a  subnet  at  the  U.S.  site,  where  a  loss  of  the  U.S.  CLBL  net  point  of 
participation  would  deny  service  to  all  other  sites.  During  MN  LOE  II,  the  U.S. 
site  lost  CLBL  net  connectivity,  denying  most  overseas  sites  access  to  the  relay 
server  and,  more  importantly,  to  the  ONADB  server.  Australia  was  not  affected. 

□  Lesson  Learned:  Proper  design  and  placement  of  servers  can  ensure  equal 
access  from  all  sites. 

□  Recommendation:  The  architecture  of  servers  for  the  experiment  should 
consider  the  underlying  network  topology,  services  required,  individual 
national  issues,  and  associated  costs. 

Title:  Training 

□  Observation:  The  training  provided  during  Week  0  was  not  sufficient  for  the  players 
even  though  prior  exposure  had  been  obtained  in  both  the  concepts  and  tools  used. 

□  Discussion:  The  training  provided  during  Week  0  suffered  from  these  major 
problems: 

o  Method  of  delivery,  using  Groove,  and  poor  audio  quality 
o  Lecture  style  of  presentations 
o  Inability  to  engage  with  the  training/lecturing  staff 
o  Lack  of  training  materials 

o  Lack  of  a  training  vignette  as  part  of  the  experiment. 
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□  Lesson  Learned:  New  concepts  and  using  technologies  with  which  the 
operators  are  unfamiliar  require  an  investment  in  the  training  of  the  operators. 
Training  while  experimenting  is  not  acceptable. 

□  Recommendation:  Invest  in  a  training  program  and  schedule. 

Title:  ONA  Geographic  Visualization  Tool 

□  Observation:  The  ONA  geographic  visualization  tool  requires  further 
refinement  in  order  to  effectively  display  data  from  the  ONA  database.  The 
geographic  display  of  infonnation  contained  in  the  ONA  database  was 
inadequate  for  an  ONA.  Because  government-produced  charts  were  not 
available  for  use,  the  display  did  not  meet  the  limited  standards  established  in 
the  pre-experiment  planning  meetings. 

□  Discussion:  The  digital  maps  did  not  contain  current  geophysical  or  cultural 
information  in  the  detail  and  scales  necessary  for  system-of-systems  analysis  or 
operational  planning.  Within  the  context  of  this  experiment,  the  exact  charts  of 
the  area  of  interest  were  not  released.  However,  symbols  used  on  the  maps  to 
indicate  the  location  of  nodes  did  not  follow  standard  military  or  cartography 
use  and  did  not  include  a  key  to  explain  the  symbols,  which  were  different 
from  those  agreed  upon  at  pre-experiment  planning  meetings.  Standardization 
was  needed  for  the  symbols  used. 

□  Lesson  Learned:  The  foundation  of  the  ONA  database  is  reliable,  current,  and 
accurate  information,  regardless  of  fonn  (words,  pictures,  charts,  maps).  This 
information  is  needed  for  the  preparation  and  use  of  the  ONA  database. 
Symbols  must  confonn  to  national  standards,  and  a  legend  should  be  provided. 

□  Recommendations: 

o  The  users  of  the  ONA  database  should  meet  to  establish  requirements  for 
the  graphic  display. 

o  Use  standard  symbols  for  annotating  the  graphic  visualization  tool,  and 
provide  a  legend. 

o  JFCOM  J2  and  J9  personnel  should  work  with  NIMA  to  ensure  that  current 
and  accurate  digital  charts  and  maps  are  available  on  demand  for  all  parts 
of  the  world. 

Title:  Replication  or  Modeling  of  Future  Technologies 

□  Observation:  Groove  was  unable  to  support  all  of  the  functionality  necessary 
to  replicate  the  CIE  capability. 

□  Discussion:  Experimentation  required  current  technology  in  order  to  replicate 
the  capabilities  of  technologies  of  the  future.  If  that  were  truly  possible,  they 
wouldn’t  be  technologies  of  the  future,  but  rather,  technologies  of  today. 
However,  the  characteristics  of  a  projected  future  technology  may  be  replicated. 

o  What  does  the  technology  of  the  future  enable  us  to  do  that  we  cannot  do 
today? 

o  How  can  those  characteristics  be  replicated? 
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o  Must  all  of  the  characteristics  of  the  future  be  in  one  package  or  resemble 
the  technology  of  the  future?  Indeed,  maybe  the  solution  is  to  use  more 
basic,  but  proven,  capabilities  that  can  accurately  emulate  characteristics 
desired.  Otherwise,  experiments  are  conducted  within  experiments.  If  one 
fails,  then  it  has  the  potential  to  affect  the  other  negatively. 

□  Lesson  Learned:  Stick  with  basic,  but  proven,  capabilities  for  large 
collaborative  groups  that  can  accurately  emulate  characteristics  desired  for  the 
projected  future  technologies. 

□  Recommendation:  Identify  those  characteristics  of  technology  that  aren’t 
currently  available,  then  determine  and  implement  tested  and  true  techniques 
for  replicating  them. 

Title:  Setup  of  Participant  Accounts 

□  Observation:  Technical  requirements  for  participant  setup  actions  must  be 
linked  directly  with  the  manning  document. 

□  Discussion:  Setting  permissions  and  access  restrictions  occurred  four  separate 
times  for  this  event.  Twenty  shared  spaces  and  several  database  pennissions 
needed  to  be  set  for  each  of  more  than  180  participants,  resulting  in  nearly 
60,000  pennissions  decisions  during  account  setup.  The  effect  of  tackling  this 
exercise  four  times  was  mind-boggling.  This  experience  can  be  directly 
attributed  to  the  ever-changing  manning  document,  all  the  way  up  to  the 
commencement  of  the  experiment.  Most  of  the  difficulties  stemmed  from 
constant  changes  in  the  player  list.  U.S.  personnel  had  to  make  their  best 
educated  guess  in  order  to  begin  the  experiment,  and  then  they  modified  the 
manning  document  as  changes  occurred.  Additionally,  a  problem  within  the 
servers  or  the  software  complicated  the  matter.  Information  was  input  but  did 
not  take  effect  or  changed  overnight  on  several  occasions. 

□  Lesson  Learned:  Technical  and  concepts  personnel  must  agree  on  the  data 
needed  for  setting  up  participant  accounts  and  must  link  that  infonnation  to  the 
master  participants’  manning  document.  Additionally,  this  task  must  be  done  in 
a  coordination  mode  rather  than  a  control  mode.  The  work  must  be  distributed. 

□  Recommendation:  Integrate  technical  requirements  needed  to  set  up  accounts 
for  participants  with  the  event-manning  document,  so  that  event  setup 
requirements  are  clear.  The  matrix  used  to  build  the  participant  list  should 
include  fields  for  individual  user  accounts  and  pennissions.  In  past 
experiments,  the  information  manager  performed  this  key  integration. 

Title:  ClearResearch  as  a  Tool  for  System-of-Systems  Analysis 

□  Observation:  To  detennine  if  the  program  should  be  adopted  as  a  tool  for  the 
SoSA  portion  of  developing  an  ONA,  U.S.  SoSAs  evaluated  ClearResearch  so 
that  it  would  not  interfere  with  the  primary  objectives  of  MN  LOE  II. 

□  Discussion:  The  evaluation  plan  for  ClearResearch  called  for  selected  PMESII 
analysts,  trained  in  this  tool,  to  conduct  searches  in  order  to  create  or  update 
nodes.  The  performance  of  ClearResearch  was  evaluated  by  its  ability  to 
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create  or  update  nodes.  Immediately  after  the  experiment,  PMESII  analysts 
used  ClearResearch  to  create  nodes  that  could  have  been  added  to  the  database 
during  the  experiment,  creating  a  larger  sample  size  for  the  evaluation. 
Information  provided  by  ClearResearch  was  evaluated  using  a  survey  created 
by  the  PMESII  analysts.  The  total  sample  size — during  and  after  the 
experiment — was  74.  Survey  responses  from  the  PMESII  analysts  showed  that 
the  relationship  or  category  map  was  not  useful  37.5  percent  of  the  time. 

□  Lesson  Learned:  Based  on  this  limited  trial,  ClearResearch  did  not  prove  to  be 
sufficiently  dependable  or  useful  to  PMESII  analysts  who  conducted  the 
specific  task  of  research  for  the  SoSA  portion  of  developing  an  ONA. 

□  Recommendation:  More  research  is  required  on  ClearResearch  in  order  to 
fully  evaluate  it.  Additionally,  alternative  options  for  SoSA  research  tools 
should  be  explored. 

INFORMATION  SHARING 

Title:  Linking  the  MNIS  OPCON  to  SJFHQ 

□  Observation:  In  the  USJFCOM  J6  operational  concept,  “Multinational 
Infonnation  Sharing  for  Allies  and  Coalition  Members,”  the  CISP3  is 
introduced  as  an  integral  part  of  the  standing  joint  force  headquarters 
component  of  any  national  military  force.  However,  the  document  does  not 
address  how  the  CISP3  is  integrated  into  the  SJFHQ.  Nor  does  the  paper 
consider  alternative,  non-U. S. -based  polices  and  procedures  for  sharing 
information  in  multinational  operations. 

□  Discussion:  Is  the  CISP3  a  board,  a  center,  or  a  cell,  and/or  how  does  it  fit  into 
the  operations/planning  process?  Additionally,  no  membership  is  described  for 
the  CISP3.  Does  it  include  members  from  all  the  SJFHQ  groups,  such  as 
operations,  plans,  and  information  superiority?  The  J6  document  also  heavily 
implies  a  foreign  disclosure  role  within  the  CISP3.  Under  the  current  manning 
roster  of  the  SJFHQ,  no  foreign  disclosure  officer  billets  exist.  The  MNIS 
document  also  does  not  adequately  describe  foreign  disclosure  roles  and 
responsibilities  within  the  CISP3. 

□  Lessons  Learned:  The  CISP3  OPCON  must  be  revised. 

□  Recommendation:  USJFCOM  J6  must  revisit  and  refine  the  description,  roles, 
and  responsibilities  of  the  CISP3  and  must  explain  how  the  CISP3  becomes  an 
integral  part  of  the  SJFHQ. 

Title:  U.S.  National  Disclosure  Policy  (NDP-1) 

□  Observation:  U.S.  policy  NDP-1  does  not  afford  commanders  the  flexibility 
necessary  to  share  information  during  pre-crisis. 

□  Discussion:  The  policy  is  intended  to  be  restrictive;  thus,  the  U.S.  shares 
information  by  exception.  Coalition  information  sharing  requires  a  policy  that 
withholds  information  by  exception.  Information  sharing  in  the  context  of 
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effects-based  operations  exists  on  two  plains:  blue  forces’  view  of  red  forces, 
or  infonnation  about  the  adversary,  and  blue  forces’  view  of  blue  forces,  or 
information  about  one’s  own  capabilities  and  force  disposition.  Information 
about  blue  force  capabilities  is  controlled  by  the  Secretary  of  Defense  and 
operational  commanders  and  carries  significant  operational  security  and  force 
protection  implications.  Policies  enforced  during  actual  operations  also  must 
apply  to  pre-crisis  situations  and,  to  a  certain  extent,  to  experimentation. 
Currently,  none  of  the  policies  address  experimentation,  thus  making  the 
process  extremely  difficult. 

□  Lessons  Learned:  The  current  sharing  policy  is  not  compatible  within  a 
multinational  collaborative  environment.  Both  of  the  information-sharing 
realms  must  be  addressed,  demanding  high-level  coordination  with  the 
intelligence  and  military  operations  communities. 

□  Recommendation:  To  acknowledge  the  information  age  and  the  necessity  to 
share  rather  than  withhold  infonnation,  request  a  serious,  detailed  examination 
of  NDP-1  and  associated  implementing  directives. 

COLLABORATION 

Title:  Experiment  Constraints 

□  Observation:  Experiment  design  and  time  constraints  limited  awareness  of  the 
complexity  of  the  ONA  process  and  allowed  non-database  means  for 
conducting  the  ONA.  The  experiment  constraints — consideration  of  only  five 
nodes  per  effect  and  time  limit — restricted  the  completeness  of  the  analysis. 

□  Discussion:  These  limitations  did  not  give  participants  an  appreciation  of  the 
potential  complexity  and  richness  of  the  analysis.  Experiment  constraints  also 
allowed  the  use  of  non-database  tools  to  propose  linkages  and  to  present  the 
results.  To  overcome  this  limitation,  the  U.S.  SoSA  team  suggested  that  the 
fourth  vignette  be  replaced  with  the  detailed  ENAR  linkage  of  a  single  directed 
effect  by  all  participants.  This  allowed  analysts  and  planners  to  examine  nodes 
from  all  PMESII  categories,  to  fully  explore  linkages  and  all  actions  in  all 
phases,  and  to  use  a  wide  range  of  national  resources. 

□  Lesson  Learned:  Experiment  and  time  constraints  limited  the  ability  of 
participants  to  fully  use  the  ONA  process  and  ONA  database  and  tools. 

□  Recommendation:  Future  experiments  should  provide  participants  with  a 
large,  detailed  ONA  and  sufficient  work  hours  to  complete  the  analysis.  This 
would  drive  participants  to  use  ONA  database  capabilities  to  create  the 
linkages.  The  ONA  graphical  layout  tool  should  be  used  to  view  and  present 
the  linkages,  nudging  ONA  participants  away  from  non-database  procedures  to 
those  using  an  automated  ONA  database. 

Title:  CIE  Concept 

□  Observation:  Even  though  the  CIE  concept  was  not  tested  within  this  LOE,  it 
was  still  the  subject  of  observations  resulting  from  the  collaborative 
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environment  used  to  develop  the  ONA.  U.S.  senior  concept  developers 
(SCDs)  suggested  that  the  CIE  concept  fails  to  address  the  knowledge  and 
leadership  aspects  of  ONA  and  EBO. 

□  Discussion:  SCDs  noted  the  failures  of  the  CIE  concept  in  detail.  A  good 
definition  of  collaboration  does  not  exist,  nor  does  a  clear  understanding  of  it. 
The  concept  addresses  the  common  relevant  operational  picture,  joint 
interactive  planning,  and  the  global  information  grid,  but  it  does  not  illustrate 
how  these  components  build  knowledge.  Further,  the  different  demands  that 
CIE  places  on  leadership  require  greater  emphasis. 

□  Lesson  Learned:  The  CIE  concept  needs  further  work. 

□  Recommendation:  Reemphasize  and  rewrite  the  CIE  white  paper  to 
incorporate  the  lessons  learned  from  this  LOE,  with  an  emphasis  on  the 
knowledge  and  leadership  aspects  of  CIE. 

EXPERIMENT  DESIGN 

Title:  Leveraging  Other  Multinational  Endeavors 

□  Observation:  Some  of  the  challenges  of  this  experiment  have  been  explored  by 
the  joint  warrior  interoperability  demonstration  (JWID). 

□  Discussion:  While  not  an  experimentation  effort,  JWID  addressed  the 
multinational  collaborative  tool  issue.  Artificial  geographies  and  histories  have 
been  created  for  JWID.  Additionally,  most  of  those  involved  in  the 
multinational  experimentation  effort  participate  in  JWID.  In  the  past,  JWID 
has  been  nothing  more  than  a  technology  demonstration  where  vendors  display 
their  wares  and  had  nothing  to  do  with  future  combat  capabilities.  J9  reviewed 
the  JWID  03  plan  and  detennined  they  would  not  participate.  JWID  04  may 
offer  another  opportunity  to  explore;  however,  the  event  is  focused  on 
Northern  Command  and  therefore  may  not  be  valuable  for  multinational 
partners.  J9  remains  in  contact  with  JWID  organizers. 

□  Lesson  Learned:  Much  of  the  work  from  JWID  and  other  multinational 
endeavors  may  reduce  manpower  efforts  on  this  end,  as  well  as  training 
requirements  on  the  other  end. 

□  Recommendation:  In  the  future,  explore  multinational  endeavors,  including 
JWID,  to  identify  elements  to  be  used  for  experimentation  efforts. 

Title:  Requirements  Collection  and  Tool  Selection 

□  Observation:  The  selection  of  tools  for  MN  LOE  II  did  not  result  from  the 
collection  of  requirements  to  support  the  operational  concepts. 

□  Discussion:  The  tools  were  selected  to  explore  fully  the  capabilities  of  peer-to- 
peer  technology.  However,  the  lack  of  an  effort  to  collect  and  develop 
requirements  to  support  operational  concept  implementation  resulted  in 
insufficient  tool  capabilities  to  conduct  the  process  effectively.  Thus, 
collaboration  capability  and  ONA  database  functionality  were  not  optimal. 
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□  Lesson  Learned:  Requirements  must  be  defined  prior  to  tool  selection,  so  that 
essential  capabilities  can  be  addressed  and  problems  can  be  resolved  or 
avoided  during  process  employment. 

□  Recommendation:  Future  experiment  efforts  should  demand  the  collection  of 
requirements  prior  to  tool  selection  in  order  to  support  operational  concept 
employment  functionality. 

Title:  Role  of  Liaison  Officers  (LNOs)  in  Experiment  Planning  and  Execution 

□  Observation:  LNOs  solved  or  prevented  several  problems,  decreasing  the 
overall  demand  on  technical  and  control  elements. 

□  Discussion:  Since  habitual  relationships  breed  familiarity,  trust,  and  cohesion, 
a  single  point  of  contact  ensures  that  all  of  a  nation’s  concerns  are  addressed. 
Additionally,  by  selecting  LNOs  with  diverse  backgrounds  and  expertise,  a  de 
facto  alternative  experiment  control  was  created.  Moreover,  LNOs  who  were 
engaged  throughout  the  planning  process  were  more  likely  to  identify  the 
correct  problem  solver.  An  LNO’s  greatest  value  is  to  serve  as  a  translator  of 
sorts — to  be  able  to  say,  “Here’s  what  they  said,  now  here’s  what  they  meant.” 

□  Lesson  Learned:  The  presence  of  LNOs  throughout  the  planning  and 
execution  phase  is  critical.  LNOs  with  a  diverse  background  who  have  been 
cross-trained  are  most  useful. 

□  Recommendation:  Continue  to  use  LNOs  in  multinational  experimentation 
efforts.  Select  the  LNOs  early  in  the  process  to  maximize  the  time  needed  for 
familiarity,  trust,  and  cohesion  to  grow.  Select  LNOs  with  diverse  backgrounds, 
including  technical  and  conceptual  backgrounds,  and  cross-train  them  to  the 
maximum  extent  practicable.  Also,  consider  extending  the  LNO  concept  to 
allow  an  exchange — rather  than  U.S.  placement — of  LNOs  in  future  events. 
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Title:  Information  Manager  Usage 

□  Observation:  An  infonnation  manager  would  have  helped  immensely  in 
preparing  and  executing  this  event.  The  operational  net  assessment  and 
coalition  information-sharing  processes  were  hampered  due  to  a  lack  of 
knowledge  management  principles  in  their  processes. 

□  Discussion:  In  past  events,  an  information  manager  kept  a  log  of  significant 
events,  kept  the  event  rolling,  and  generally  assisted  the  chief  controller  with 
event  organization.  During  pre-event  operations,  the  infonnation  manager 
would  coordinate  setup  details,  would  assist  the  chief  controller  with 
contingencies,  and  would  interface  with  other  key  players.  A  knowledge  or 
information  management  plan  would  have  greatly  reduced  the  general 
confusion  among  the  experiment  participants  about  document  management, 
collaboration  processes,  and  infonnation- sharing  practices. 

□  Lesson  Learned:  An  information  manager  enhances  all  elements  of  execution. 
Lack  of  knowledge  managers  and  of  a  knowledge  management  plan  will 
inhibit  operational  process  refinement  experiments. 

□  Recommendation:  Knowledge  management  personnel  must  participate  with  the 
national  teams  and  a  knowledge  management  plan  must  exist  for  future 
experiments.  The  infonnation  manager  should  be  a  required  position  for  all  events. 

Title:  Interagency  Participants 

□  Observation:  Interagency  participants  are  needed  to  fully  examine  the  ONA 
process.  Most  participants  from  multiple  countries  were  affiliated  with  the  military. 

□  Discussion:  Lack  of  interagency  participants  limited  the  consideration  of 
diplomatic  and  political  actions  and  resources.  Most  military  members  do  not 
have  the  breadth  of  experience  in  nonmilitary  agencies  nor  the  capability  to 
identify  nomnilitary  aspects  of  the  ONA.  Participant  nations’  militaries  have  been 
working  together  so  closely  and  for  so  long  that  a  significant  cultural  difference 
may  not  exist  in  the  group’s  professional  approach  to  developing  the  ONA.  The 
cultural  variances  among  military  personnel  from  different  nations  may  be  less 
pronounced  than  those  between  military  and  political  personnel  within  nations. 

□  Lesson  Learned:  Solely  military-affiliated  experiment  participants  do  not 
provide  a  sufficient  breadth  of  knowledge  or  experience  to  fully  perfonn  an 
ONA. 

□  Recommendation:  In  order  to  fully  build  an  ONA,  interagency  participants  are 
required  from  each  of  the  countries  involved. 

Title:  Release/Disclosure  of  Geospatial  Information  and  Services  Products 

□  Observation:  Several  obstacles  were  encountered  in  obtaining  authorization  to 
disclose  or  release  National  Imagery  and  Mapping  Agency  (NIMA)  data  for 
the  MN  LOE  II  play  box. 
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□  Discussion:  In  the  planning  phases  of  MN  LOE  II,  the  play  box  was  identified 
in  order  to  begin  crafting  the  scenario  and  building  knowledge  of  the  area. 
Since  the  experiment  was  a  multinational  event,  appropriate  geospatial 
information  and  services  were  available  and  releasable  to  the  multinational 
participants.  Ultimately,  only  some  products  could  be  released  to  multinational 
participants;  however,  those  released  were  sufficient  to  conduct  the 
experiment.  Within  the  process,  a  minimum  of  information  was  needed  to 
submit  a  request  for  disclosure  or  release  authorization: 

o  Purpose:  Will  the  products  be  used  for  exercise  or  experiment? 

o  Participants:  Identify  the  multinational  participants  and  observers  who  will 
require  access  to  the  products. 

o  Mapping  Products:  Identify  the  sheet  numbers  and  series,  or  other  GIS 
product  data,  including  the  specific  scale. 

o  Timeline:  Identify  the  time  frame  of  the  event.  Consider  planning  conferences 
and  pre-event  workshops  in  determining  when  the  infonnation  is  sought. 

o  Release  or  Disclosure:  Specifically,  will  the  products  be  released,  or 
“given,”  to  the  multinational  participants  to  keep,  or  will  they  be  simply 
disclosed,  or  “displayed,”  to  the  participants? 

o  Product  Control 

□  Lesson  Learned:  A  detailed  request  for  disclosure  or  release  must  be  submitted 
early  in  the  planning  phases.  Linal  authorization  to  disclose  and/or  release 
NIMA  limited-distribution  data  is  the  responsibility  of  the  NIMA  disclosure 
and  release  office.  In  addition,  multinational  event  planners  must  maintain 
close  and  continuing  coordination  among  themselves,  the  USJLCOM  foreign 
disclosure  officer,  and  the  command’s  NIMA  liaison  officer. 

□  Recommendation:  A  coordination  effort  for  release  of  NIMA  limited- 
distribution  data  should  begin  immediately  after  the  scenario  play  box  has  been 
identified,  after  the  initial  planning  conference.  Additionally,  frequent 
communication  is  required  with  agencies  and  offices  involved  in  the  request 
process  to  reconcile  the  status  of  any  requests. 

Title:  Classification  Guidance 

□  Observation:  The  discussion  and  decision  regarding  classification  guidance  for 
MN  LOE  II  was  a  long,  involved  process  that  required  insights  and  input  from 
all  of  the  multinational  partners. 

□  Discussion:  While  infonnation  related  to  the  scenario  was  classified,  the  actual 
information  used  to  construct  the  ONA  database  and  the  attendant  scenario  was 
unclassified.  Accordingly,  the  risk  of  inadvertent  release  or  disclosure  or 
possible  security  violations  was  minimal,  resulting  in  no  urgency  to  decide 
classification  guidance  before  research  began  for  the  ONA  database.  In 
subsequent  and  succeeding  experiments,  these  artificialities  may  not  be 
present.  In  reality,  classification  guidance  should  be  promulgated  to  all 
participants  prior  to  database  and  scenario  development.  New  ONA  database 
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development  from  scratch  is  time-consuming  and  manpower-intensive,  and 
requires  much  lead-time. 

□  Lesson  Learned:  Given  the  prolonged  ONA  database  development, 

classification  guidance  must  be  detennined  early  in  the  planning  process — by 
the  concept  development  conference,  if  possible — to  enable  work  to  commence 
on  the  scenario  and  database. 

□  Recommendation:  Develop  and  promulgate  to  all  participants  standard 
classification  guidance  for  the  experiment  early  in  the  planning  process,  and  do 
not  deviate  from  that  guidance,  except  under  extreme  circumstances. 

Title:  Senior  Concept  Developer  (SCD)  Pre-Event  Caucus  and  Orientation 

□  Observation:  Use  of  SCDs  who  met  prior  to  the  LOE  to  address  main 
objectives  and  associated  issues  contributed  significantly  to  the  LOE  breadth. 

□  Discussion:  The  insightful  opinions  of  SCDs  are  a  proven  added  value  to  any 
experiment  on  military  concepts,  especially  when  introducing  U.S.  concepts  to 
a  multinational  audience.  Bringing  the  SCDs  together  for  a  face-to-face 
orientation  allowed  them  to  focus  on  the  right  issues  during  the  LOE.  Since 
they  had  met  prior  to  the  event  and  had  been  introduced  to  the  LOE 
background  and  objective,  the  SCDs  were  more  candid  in  their  comments 
during  the  azimuth  checks  and  in-focus  sessions. 

□  Lesson  Learned:  SCD  pre-event  caucus  and  orientation  significantly  increase 
the  contributions  of  SCDs. 

□  Recommendation:  Plan  an  SCD  pre-event  caucus  and  orientation  to  focus  the 
SCDs  on  the  event  objectives  and  issues. 
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APPENDIX  I.  ACRONYMS  AND  ABBREVIATIONS 


AAR  -  After  Action  Review 

ARL  -  Army  Research  Laboratory 

AUS  -  Australia 

BL1  -  Bilateral  1 

BL2  -  Bilateral  2 

CAN  -  Canada 

CBIS  -  Content  Based  Information 
System 

CC  -  Control  Cell 

CCEB  -  Combined  Communications- 
Electronics  Board 
CCIB  -  Command  and  Control 
Interoperability  Board 
CD  -  compact  disk 
CDC  -  Concept  Development 
Conference 

CDE  -  Concept  Development  and 
Experimentation  (NATO  SACLANT) 
CDR  USPACOM  -  Commander,  United 
States  Pacific  Command 
CFBLNet  -  Combined  Federated  Battle 
Laboratories  Network 
CFEC  -  Canadian  Forces 
Experimentation  Centre 
C4I  -  Command  Control 
Communications  Computer  and 
Information 

CIE  -  Collaborative  information 
Environment 

CISP3  -  Coalition  Information-Sharing 
Policy  and  Procedures  Panel 
COA  -  Course  of  action 
COI  -  Critical  Operational  Issue 
CONOPS  -  Concept  of  Operations 
CWC  -  Chemical  Weapons  Convention 
DIME  -  Diplomatic,  Infonnation, 
Military,  Economic 
DOTMLPF  -  Doctrine,  Organization, 
Training,  Materiel,  Leadership 
Development,  Personnel,  and  Facilities 
DSTL  -  Defence  Science  &  Technology 
Laboratory 

DSTO  -  Defence  Science  &  Technology 
Organization 


EBO  -  Effects  Based  Operations 
EEZ  -  Exclusive  Economic  Zone 
ENAR  -  Effects,  Nodes,  Actions, 
Resources 

EU  -  European  Union 
FDO  -  Foreign  Disclosure  Officer 
FFG  -  Guided  missile  frigate 
GER  -  Germany 
GL  -  Graphical  Layout 
GT  -  Gross  tons 
HF  -  Human  Factors 
IDP  -  Internally  displaced  persons 
IM  -  Infonnation  Manager 
IOM  -  International  Organization  for 
Migration 

IP  -  Internet  Protocol 
IS  -  Information  sharing 
JBC  -  Joint  Battle  Center 
JDCAT  -  JBC  Data  Collection  and 
Analysis  Tool 

JIACG  -  Joint  Interagency  Control 
Group 

JROC  -  Joint  Requirements  Oversight 
Council 

JWICS  -  Joint  Worldwide  Intelligence 
Communications  System 
JWID  -  Joint  Warrior  Interoperability 
Demonstration 
KM  -  Knowledge  Manager 
LNO  -  Liaison  officer 
LOCE  -  Linked  Ops-Intel  Centers 
Europe 

LOE  -  Limited  Objective  Experiment 
Mbps  -  Megabytes  per  second 
MIC  -  Multinational  Interoperability 
Council 

ML  -  Multilateral 
MLS  -  Multi-Level  Security 
MN  -  Multinational 
MNC  -  Multinational  Corporation  / 
Multinational  Conglomerate 
MNIS  -  Multinational  Information 
Sharing 
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MNISACP  -  MN  Information  Sharing 
for  Allies  and  Coalition  Partners 
MOA  -  Memorandum  of  Agreement 
MOU  -  Memorandum  of  Understanding 
MSEL  -  Master  Scenario  Events  List 
NATO  -  North  Atlantic  Treaty 
Organization 

NDP  -  National  Disclosure  Policy 
NGO  -  Non-governmental  organization 
NIMA  -  National  Imagery  and  Mapping 
Agency 

NIPRNET  -  Non-secure  Internet 
Protocol  Router  Network 
NM  -  Nautical  mile 
NSA  -  National  Security  Agency 
ONA  -  Operational  Net  Assessment 
ONA  GL  -  Operational  Net  Assessment 
Graphical  Layout  tool 
OPEC  -  Organization  of  Petroleum 
Exporting  Countries 
OPCON  -  Operational  Concept 
PMESII  -  Political,  Military,  Economic, 
Social,  Infrastructure,  Information 
POP  -  Point  of  Participation 
P2P  -  Peer  to  Peer 
RF  -  radio  frequency 
SA  -  Situational  Awareness 
SACLANT  -  Supreme  Allied 
Commander  Atlantic 
SCD  -  Senior  Concept  Developer 


SIPRNET  -  Secret  Internet  Protocol 
Router  Network 
SJFHQ  -  Standing  Joint  Force 
Headquarters 

SLOC  -  Sea  Lines  of  Communication 
SME  -  Subject  Matter  Expert 
SoSA  -  System-of-System  Analyst 
SPSS  -  Statistical  Product  and  Service 
Solutions 

SQL  -  Structured  Query  Language 
SWA  -  Southwest  Asia 
SPAWAR  -  Space  and  Naval  Warfare 
Systems  Command 
TL  -  Trilateral 

TTPs  -  Tactics,  Techniques,  and 
Procedures 
U  -  Unclassified 
UK  -  United  Kingdom 
UNHCR  -  UN  High  Commissioner  on 
Refugees 

US  -  United  States 

USJFCOM  -  United  States  Joint  Forces 
Command 

VIP  -  Very  Important  Person 
WAN  -  Wide  Area  Network 
WS  -  workshop 

ZULU  -  Time  zone  indicator  for 
Universal  Time 
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APPENDIX  J.  CONCEPT  OVERVIEWS 


Operational  Net  Assessment 

Operational  net  assessment  (ONA)  is  the  integration  of  people,  processes,  and 
tools  that  use  multiple  infonnation  sources  and  collaborative  analysis  to  enhance 
command  decision-making.  This  continuous,  dynamic  process  produces  a  coherent, 
relevant,  and  shared  knowledge  environment,  as  well  as  supporting  tools,  for  planners  and 
decision-makers  to  focus  warlighting  capabilities. 

In  addition,  ONA  is  the  catalyst  for  battlespace  understanding  that  enables  decision 
superiority — the  effective  and  timely  decisions  that  ensure  mission  success.  ONA  uses 
collaboration  technologies  and  subject  matter  expertise  to  transform  data  into  useful 
information,  which,  in  turn,  becomes  actionable  knowledge. 

With  the  ONA  concept,  U.S.  Joint  Forces  Command  is  leading  the  effort  to  build 
the  knowledge  environment  our  national  leaders  and  multinational  partners  need  to 
execute  an  effective  strategy  that  employs  all  elements  of  our  combined  power. 

The  ONA  process  frames  our  understanding  of  a  potential  adversary’s  political, 
military,  economic,  social,  infonnation,  and  infrastructure  (PMESII)  systems.  Link  analysis, 
network  analysis,  and  structured  argumentation  are  used  to  assess  the  adversary’s  systems. 

Such  systems  analysis: 

□  Reveals  critical  nodes  and  vulnerabilities  that  can  be  used  in  effects-based 
operations 

□  Recognizes  the  adversary’s  goals,  intentions,  strengths,  weaknesses,  and 
behaviors 

□  Generates  understanding  and  knowledge  that  may  be  used  to  predict  indirect 
and  unintended  effects  a  of  diplomatic,  information,  military,  or  economic 
applications  of  national  power 

□  Determines  what  the  adversary  values  most  and  how  to  affect  it  decisively. 

Today’s  warfighting  commanders  receive  joint  intelligence  that  focuses  almost 
entirely  on  military  options  to  attack  an  enemy’s  warfighting  potential.  Alternatively, 
ONA  offers  a  new,  more  extensive  way  to  view  an  adversary;  it  broadens  the  scope  of  a 
commander’s  analysis  to  the  totality  of  an  adversary.  ONA  allows  the  United  States  and 
coalition  partners  to  understand  the  systemic  linkages  that  sustain  an  adversary’s  behavior 
and  ability  to  continue  the  fight. 

ONA  fosters  analysis  of  the  adversary  as  a  system  of  systems;  understanding  of  key 
relationships,  dependencies,  and  vulnerabilities;  and  identification  of  means  to  influence 
capabilities,  perceptions,  decision-making,  and/or  behavior. 
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Since  the  Department  of  Defense  alone  cannot  generate  the  information  required  to 
support  ONA  development,  national  agencies  and  foreign  governments  will  work  together 
to  develop  ONA. 

While  the  ONA  process  is  integrated  within  the  theater’s  Unified  Command 
headquarters,  it  also  takes  advantage  of  an  international  network  of  “centers  of  excellence” 
by  using  its  high-volume  communications  systems. 

This  network  will  give  the  theater  commanders  access  to  the  full  capabilities  of  the 
U.S.  interagency  community,  as  well  as  to  nongovernmental  and  allied  and  coalition 
partners.  All  are  significant  partners  who  have  information  to  contribute  and  engagement 
capabilities  to  consider. 

From  peacetime  through  conflict,  ONA  development  is  a  continuous  process: 

□  By  increasing  our  understanding  of  a  potential  enemy  and  the  conditions  of  our 
relationship  with  this  party,  ONA  may  contribute  to  achieving  our  national 
strategic  and  operational  objectives. 

□  By  using  our  full  range  of  national  capabilities,  we  avoid  conflict  by  engaging 
a  competitor  or  opponent  in  the  influence-and-deter  phases  of  the  relationship. 

□  However,  if  influence-and-deter  operations  fail  to  achieve  our  national 
objectives,  and  conflict  is  imminent,  then  ONA  describes  a  set  of  “defeat 
mechanisms”  to  accomplish  our  objectives  rapidly  and  decisively,  as  well  as  to 
deny  the  enemy  the  ability  to  accomplish  his. 
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Multinational  Information  Sharing 

The  multinational  infonnation  sharing  (MNIS)  concept  envisions  the  sharing  of 
information  in  a  single  environment.  Within  this  environment,  information  is  protected  at 
its  source,  and  access  control  is  based  upon  the  participants’  authorizations.  Not  all 
participants  will  have  access  to  all  information.  Factors  such  as  country  and  classification 
level  may  be  used  for  controlling  access.  This  environment  will  support  the  processing, 
storing,  and  transmission  of  releasable  information  from  prehostilities  through  post¬ 
combat  operational  planning  and  execution. 

Only  participants  of  the  multinational  operation  are  allowed  access  within  the 
multinational  information-sharing  environment,  and  access  to  selected  information,  based 
upon  their  credentials.  The  operational  concept  introduces  a  need  for  a  Coalition 
Infonnation-Sharing  Policy  and  Procedures  Panel  (CISP3)  as  an  integral  part  of  a  standing 
joint  force  headquarters  component  of  any  national  military  force.  The  CISP3  will  not 
circumvent  national  disclosure  policies  of  member  nations  but  will  serve  as  an  arbiter  of 
those  policies.  Its  composition  must  include  liaison  officers  working  with  the  national 
military  force,  and  it  will  help  to  develop,  establish,  and  articulate  the  policy  and 
procedures  by  which  member  nations  will  conduct  operations. 

System-of-Systems  Analyses 

The  ONA  frames  our  understanding  of  a  potential  adversary  through  political, 
military,  economic,  social,  information,  and  infrastructure  (PMESII)  systems  analyses. 
The  systems  analyses  survey  the  adversary  as  a  whole  and  then  seek  relationships  within 
and  among  systems,  as  well  as  relevant  details  in  each  subsystem.  Techniques  such  as 
link  analysis,  structured  argumentation,  and  network  analysis  depict  the  adversary’s 
systems.  The  systems  analyses: 

□  Reveal  critical  nodes  and  vulnerabilities  that  can  be  used  in  effects-based 
operations 

□  Identify  the  relationships  among  important  aspects  of  the  adversary’s  social 
fabric 

□  Identify  effects  that  will  influence  decision-making,  will,  and  behavior 

□  Promote  understanding  of  the  adversary’s  goals,  intentions,  and  perceived 
strengths  and  weaknesses 

□  Provide  knowledge  that  can  be  used  to  predict  second-  and  third-order  and 
unintended  effects,  as  a  result  of  diplomatic,  infonnation,  military,  or  economic 
applications  of  national  power 

□  Detennine  how  to  affect  efficiently  and  decisively  what  the  adversary  values  most. 

This  type  of  analysis  requires  a  thorough  understanding  of  the  adversary’s  culture, 
social  influences,  government,  religion,  and  other  intangible  factors  that  are  not  available 
in  traditional  intelligence  estimates.  The  analysis  supports  tangible  understanding  of  the 
adversary,  such  as  the  location  of  traditional  military  targets,  lines  of  communication, 
political  organizational  structure,  and  infrastructure  targets.  It  also  supports  intangible 
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understanding  of  cultural  influences,  business  relationships,  historical  influences, 
interpersonal  relationships,  and  the  flow  of  ideas  and  attitudes.  The  information  necessary 
to  develop  this  level  of  understanding  is  drawn  from  a  wide  variety  of  sources,  including 
intelligence,  academia,  industry,  and  the  public  domain.  The  ONA  development  process 
begins  before  a  crisis  in  order  to  establish  the  baseline  of  knowledge  and  understanding 
required  in  order  to  conduct  actions. 
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INITIAL  DISTRIBUTION  LIST 


1.  United  States: 

OUSD 

Joint  Staff  (J6) 

DISA 

USEUCOM/J6 

USECENTCOM/J6 

USNORTHCOM/J6 

USPACOM/J6 

USSTRATCOM/J6 

UNTRANSCOM/J6 

USSOCOM/J6 

USFORCE  KOREA/J6 

USSOUTHCOM/SCJ6 

USJFCOM  (Senior  Mentors-3) 

OFFICE  OF  HOMEFAND  SECURITY 
(CIO) 

Joint  Experimentation,  J9 
US  Joint  Forces  Command 
1562  Mitscher  Avenue 
Norfolk,  VA  23551-2488 

J2,  Attn:  Dave  Pope 

1562  Mitscher  Avenue,  Suite  200 

Norfolk,  VA  23551-2488 

J5,  Attn:  Dr.  Brian  Walsh 
1562  Mitscher  Avenue,  Suite  200 
Norfolk,  VA  23551-2488 

J6,  Attn:  CDR  Berke  J614 
1562  Mitscher  Avenue,  Suite  200 
Norfolk,  VA  23551-2488 

Rear  Admiral  Ken  Slaght,  USN 

Commander,  Space  and  Naval  Warfare 

Systems  Command 

SPAWAR  Code  00 

4301  Pacific  Highway 

San  Diego,  C A  92110-3127 


Commander,  Space  and  Naval  Warfare 

Systems  Command 

Office  of  the  Chief  Engineer 

SPAWAR  Code  05 

4301  Pacific  Highway 

San  Diego,  CA  92110-3127 

Commander,  Space  and  Naval  Warfare 
Systems  Command 

Combatant  Commander  Interoperability 
Program  Office  (CIPO) 

SPAWAR  Code  054 
4301  Pacific  Highway 
San  Diego,  CA  92110-3127 

Dr.  Robin  Keesee 

Director,  ARL  Human  Research  and 
Engineering  Directorate 
Building  459 

Aberdeen  Proving  Ground,  MD  21005- 
5425 

2.  Australia: 

Brigadier  General  Richard  Wilson 
Director  General  Military  Strategy 
Department  of  Defence 
Russell  Offices 
Canberra,  ACT  2600  AUS 

WGCDR  Gavin  Small,  AUS  LNO 
Joint  Experimentation,  J9 
US  Joint  Forces  Command 
1562  Mitscher  Avenue 
Norfolk,  VA  23551-2488 

Dr.  David  Wood 

Research  Leader  Military  Systems 
Experimentation  Branch 
DSTO  Fern  Hill 
Department  of  Defence 
Canberra,  ACT  2600  AUS 
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Dr.  Paul  Whitbread 

Research  Leader  Theatre  Command 

Analysis  Branch 

DSTO  Fem  Hill 

Department  of  Defence 

Canberra,  ACT  2600  AUS 

3.  Canada: 

Canadian  Forces  Experimentation  Centre 

Shirley's  Bay  Campus 

101  Colonel  By  Drive 

Ottawa,  Ontario 

K1A0K2 

LtCol  Tony  Battista,  CA  LNO 
Joint  Experimentation,  J9 
US  Joint  Forces  Command 
1562  Mitscher  Avenue 
Norfolk,  VA  23551-2488 

4.  Germany: 

Lt.  Jens  Romer  and  Dr.  Alexander  Ritter 
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